{"record":{"id":"e292d946a98f740a","repo":"BerriAI/litellm","slug":"hosted-vllm-does-not-support-max-chunks-per-doc","errorCode":null,"errorMessage":"Hosted VLLM does not support max_chunks_per_doc","messagePattern":"Hosted VLLM does not support max_chunks_per_doc","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/llms/hosted_vllm/rerank/transformation.py","lineNumber":86,"sourceCode":"        self,\n        non_default_params: dict | None,\n        model: str,\n        drop_params: bool,\n        query: str,\n        documents: list[str | dict[str, Any]],\n        custom_llm_provider: str | None = None,\n        top_n: int | None = None,\n        rank_fields: list[str] | None = None,\n        return_documents: bool | None = True,\n        max_chunks_per_doc: int | None = None,\n        max_tokens_per_doc: int | None = None,\n        instruction: str | None = None,\n    ) -> dict:\n        \"\"\"\n        Map parameters for Hosted VLLM rerank\n        \"\"\"\n        if max_chunks_per_doc is not None:\n            raise ValueError(\"Hosted VLLM does not support max_chunks_per_doc\")\n\n        mapped_params: Final = OptionalRerankParams(\n            query=query,\n            documents=documents,\n            top_n=top_n,\n            rank_fields=rank_fields,\n            return_documents=return_documents,\n        )\n\n        # `instruction` is a vLLM-supported passthrough (folded into the model's\n        # chat_template_kwargs). Only forward it when explicitly set so omitting\n        # it leaves the request unchanged.\n        if instruction is not None:\n            mapped_params[\"instruction\"] = instruction\n\n        return dict(mapped_params)\n\n    def validate_environment(","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/llms/hosted_vllm/rerank/transformation.py#L68-L104","documentation":"The HostedVLLM rerank transformation maps Cohere-style rerank parameters onto vLLM's Jina-compatible /rerank API. vLLM's rerank endpoint has no notion of client-side chunking of long documents, so max_chunks_per_doc is rejected with an explicit ValueError instead of being silently ignored. This is a deliberate capability check: only query, documents, top_n, rank_fields, return_documents, and instruction are forwarded.","triggerScenarios":"Calling litellm.rerank(model='hosted_vllm/...', ..., max_chunks_per_doc=N) — including when the value is passed via the RerankParams pydantic object or a generic kwargs-passthrough wrapper that forwards all Cohere-style params.","commonSituations":"Porting existing Cohere/Azure rerank code to self-hosted vLLM; a generic multi-provider wrapper that always sends max_chunks_per_doc; long-document workflows where chunking was configured for Cohere and now break on vLLM.","solutions":["Remove max_chunks_per_doc for hosted_vllm calls (conditionally drop it when the provider is hosted_vllm)","Pre-chunk long documents client-side and pass the chunks as separate entries in documents","If chunk capping is a hard requirement, route rerank traffic to a provider that supports it (e.g. cohere) via model routing","Track vLLM releases: if a rerank chunking option appears, map it then"],"exampleFix":"# before\nlitellm.rerank(\n    model=\"hosted_vllm/jinaai/jina-reranker-v2-base-multilingual\",\n    api_base=base, query=q, documents=long_docs,\n    max_chunks_per_doc=64,\n)\n\n# after\nchunks = [c for d in long_docs for c in chunk_text(d, max_tokens=512)]\nlitellm.rerank(\n    model=\"hosted_vllm/jinaai/jina-reranker-v2-base-multilingual\",\n    api_base=base, query=q, documents=chunks,\n)","handlingStrategy":"validation","validationCode":"UNSUPPORTED_ON_VLLM = {\"max_chunks_per_doc\"}\n\ndef sanitize_vllm_rerank_params(params: dict, provider: str) -> dict:\n    if provider == \"hosted_vllm\":\n        return {k: v for k, v in params.items() if k not in UNSUPPORTED_ON_VLLM}\n    return params\n\nparams = sanitize_vllm_rerank_params(rerank_kwargs, \"hosted_vllm\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep per-provider allowlists of rerank params in one place instead of forwarding every kwarg","Document which Cohere-style params are vLLM-compatible in your project's provider notes","If you rely on chunking, route those requests to a provider that supports it rather than special-casing vLLM"],"tags":["hosted-vllm","rerank","unsupported-parameter","litellm"],"backgroundTag":"unsupported-parameter","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}