BerriAI/litellm · error · ValueError

Hosted VLLM does not support max_chunks_per_doc

Error message

Hosted VLLM does not support max_chunks_per_doc

What it means

Raised by HostedVLLM RerankConfig.map_cohere_rerank_params when the caller passes max_chunks_per_doc. LiteLLM's rerank interface is Cohere-shaped and accepts that parameter, but vLLM's rerank API does not support chunking documents, so the transformation rejects it with ValueError rather than silently ignoring it.

Source

Thrown at litellm/llms/hosted_vllm/rerank/transformation.py:86

        self,
        non_default_params: dict | None,
        model: str,
        drop_params: bool,
        query: str,
        documents: list[str | dict[str, Any]],
        custom_llm_provider: str | None = None,
        top_n: int | None = None,
        rank_fields: list[str] | None = None,
        return_documents: bool | None = True,
        max_chunks_per_doc: int | None = None,
        max_tokens_per_doc: int | None = None,
        instruction: str | None = None,
    ) -> dict:
        """
        Map parameters for Hosted VLLM rerank
        """
        if max_chunks_per_doc is not None:
            raise ValueError("Hosted VLLM does not support max_chunks_per_doc")

        mapped_params: Final = OptionalRerankParams(
            query=query,
            documents=documents,
            top_n=top_n,
            rank_fields=rank_fields,
            return_documents=return_documents,
        )

        # `instruction` is a vLLM-supported passthrough (folded into the model's
        # chat_template_kwargs). Only forward it when explicitly set so omitting
        # it leaves the request unchanged.
        if instruction is not None:
            mapped_params["instruction"] = instruction

        return dict(mapped_params)

    def validate_environment(

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Remove max_chunks_per_doc from the rerank call for hosted_vllm.
2. If you share one code path across rerank providers, strip provider-specific params per provider before calling (Cohere-only params must not reach hosted_vllm).
3. If you need document-size control, chunk documents yourself before sending, since vLLM will not chunk them.

Example fix

# before
litellm.rerank(model='hosted_vllm/bge-reranker-v2-m3', query=q, documents=docs,
               api_base=base, max_chunks_per_doc=128)
# raises ValueError: Hosted VLLM does not support max_chunks_per_doc

# after
litellm.rerank(model='hosted_vllm/bge-reranker-v2-m3', query=q, documents=docs,
               api_base=base)

VLLM_UNSUPPORTED_RERANK_PARAMS = {"max_chunks_per_doc"}

def sanitize_vllm_rerank_params(params: dict) -> dict:
    bad = VLLM_UNSUPPORTED_RERANK_PARAMS & params.keys()
    if bad:
        raise ValueError(f"params {bad} are not supported by hosted_vllm rerank")
    return params

When it happens

Trigger: Calling litellm.rerank(model='hosted_vllm/...', ..., max_chunks_per_doc=N) with any non-None value. Copying Cohere rerank options (where max_chunks_per_doc is valid) into a hosted_vllm rerank call.

Common situations: Shared rerank code that sets Cohere-specific tuning params across providers; migrating from Cohere/Jina rerank to a self-hosted vLLM reranker; the same max_tokens_per_doc/max_chunks_per_doc defaults being applied indiscriminately (note: only max_chunks_per_doc is rejected here).

Related errors

• query is required for Hosted VLLM rerank
• documents is required for Hosted VLLM rerank
• No results found in the response={response}
• Missing required fields in the result={result}
• No results found in the response={response}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/e292d946a98f740a. Report an issue: GitHub.