{"record":{"id":"e292d946a98f740a","repo":"BerriAI/litellm","slug":"hosted-vllm-does-not-support-max-chunks-per-doc","errorCode":null,"errorMessage":"Hosted VLLM does not support max_chunks_per_doc","messagePattern":"Hosted VLLM does not support max_chunks_per_doc","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/llms/hosted_vllm/rerank/transformation.py","lineNumber":86,"sourceCode":"        self,\n        non_default_params: dict | None,\n        model: str,\n        drop_params: bool,\n        query: str,\n        documents: list[str | dict[str, Any]],\n        custom_llm_provider: str | None = None,\n        top_n: int | None = None,\n        rank_fields: list[str] | None = None,\n        return_documents: bool | None = True,\n        max_chunks_per_doc: int | None = None,\n        max_tokens_per_doc: int | None = None,\n        instruction: str | None = None,\n    ) -> dict:\n        \"\"\"\n        Map parameters for Hosted VLLM rerank\n        \"\"\"\n        if max_chunks_per_doc is not None:\n            raise ValueError(\"Hosted VLLM does not support max_chunks_per_doc\")\n\n        mapped_params: Final = OptionalRerankParams(\n            query=query,\n            documents=documents,\n            top_n=top_n,\n            rank_fields=rank_fields,\n            return_documents=return_documents,\n        )\n\n        # `instruction` is a vLLM-supported passthrough (folded into the model's\n        # chat_template_kwargs). Only forward it when explicitly set so omitting\n        # it leaves the request unchanged.\n        if instruction is not None:\n            mapped_params[\"instruction\"] = instruction\n\n        return dict(mapped_params)\n\n    def validate_environment(","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/llms/hosted_vllm/rerank/transformation.py#L68-L104","documentation":"Raised by HostedVLLM RerankConfig.map_cohere_rerank_params when the caller passes max_chunks_per_doc. LiteLLM's rerank interface is Cohere-shaped and accepts that parameter, but vLLM's rerank API does not support chunking documents, so the transformation rejects it with ValueError rather than silently ignoring it.","triggerScenarios":"Calling litellm.rerank(model='hosted_vllm/...', ..., max_chunks_per_doc=N) with any non-None value. Copying Cohere rerank options (where max_chunks_per_doc is valid) into a hosted_vllm rerank call.","commonSituations":"Shared rerank code that sets Cohere-specific tuning params across providers; migrating from Cohere/Jina rerank to a self-hosted vLLM reranker; the same max_tokens_per_doc/max_chunks_per_doc defaults being applied indiscriminately (note: only max_chunks_per_doc is rejected here).","solutions":["Remove max_chunks_per_doc from the rerank call for hosted_vllm.","If you share one code path across rerank providers, strip provider-specific params per provider before calling (Cohere-only params must not reach hosted_vllm).","If you need document-size control, chunk documents yourself before sending, since vLLM will not chunk them."],"exampleFix":"# before\nlitellm.rerank(model='hosted_vllm/bge-reranker-v2-m3', query=q, documents=docs,\n               api_base=base, max_chunks_per_doc=128)\n# raises ValueError: Hosted VLLM does not support max_chunks_per_doc\n\n# after\nlitellm.rerank(model='hosted_vllm/bge-reranker-v2-m3', query=q, documents=docs,\n               api_base=base)\n","handlingStrategy":"validation","validationCode":"VLLM_UNSUPPORTED_RERANK_PARAMS = {\"max_chunks_per_doc\"}\n\ndef sanitize_vllm_rerank_params(params: dict) -> dict:\n    bad = VLLM_UNSUPPORTED_RERANK_PARAMS & params.keys()\n    if bad:\n        raise ValueError(f\"params {bad} are not supported by hosted_vllm rerank\")\n    return params","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep per-provider rerank param sets: Cohere-only knobs must be stripped before forwarding to hosted_vllm.","Prefer failing on unknown params in your wrapper (like this handler does) over silently forwarding them."],"tags":["hosted-vllm","rerank","parameters","validation","cohere-compat"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}