BerriAI/litellm · error · ValueError

Hosted VLLM does not support max_chunks_per_doc

Error message

Hosted VLLM does not support max_chunks_per_doc

What it means

The HostedVLLM rerank transformation maps Cohere-style rerank parameters onto vLLM's Jina-compatible /rerank API. vLLM's rerank endpoint has no notion of client-side chunking of long documents, so max_chunks_per_doc is rejected with an explicit ValueError instead of being silently ignored. This is a deliberate capability check: only query, documents, top_n, rank_fields, return_documents, and instruction are forwarded.

Solutions

1. Remove max_chunks_per_doc for hosted_vllm calls (conditionally drop it when the provider is hosted_vllm)
2. Pre-chunk long documents client-side and pass the chunks as separate entries in documents
3. If chunk capping is a hard requirement, route rerank traffic to a provider that supports it (e.g. cohere) via model routing
4. Track vLLM releases: if a rerank chunking option appears, map it then

Example fix

# before
litellm.rerank(
    model="hosted_vllm/jinaai/jina-reranker-v2-base-multilingual",
    api_base=base, query=q, documents=long_docs,
    max_chunks_per_doc=64,
)

# after
chunks = [c for d in long_docs for c in chunk_text(d, max_tokens=512)]
litellm.rerank(
    model="hosted_vllm/jinaai/jina-reranker-v2-base-multilingual",
    api_base=base, query=q, documents=chunks,
)

UNSUPPORTED_ON_VLLM = {"max_chunks_per_doc"}

def sanitize_vllm_rerank_params(params: dict, provider: str) -> dict:
    if provider == "hosted_vllm":
        return {k: v for k, v in params.items() if k not in UNSUPPORTED_ON_VLLM}
    return params

params = sanitize_vllm_rerank_params(rerank_kwargs, "hosted_vllm")

When it happens

Trigger: Calling litellm.rerank(model='hosted_vllm/...', ..., max_chunks_per_doc=N) — including when the value is passed via the RerankParams pydantic object or a generic kwargs-passthrough wrapper that forwards all Cohere-style params.

Common situations: Porting existing Cohere/Azure rerank code to self-hosted vLLM; a generic multi-provider wrapper that always sends max_chunks_per_doc; long-document workflows where chunking was configured for Cohere and now break on vLLM.

Related errors

• api_base must be provided for Hosted VLLM rerank
• documents is required for Hosted VLLM rerank
• Error parsing response
• Missing required fields in the result=
• No results found in the response=

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/e292d946a98f740a. Report an issue: GitHub.

Appendix: source

Thrown at litellm/llms/hosted_vllm/rerank/transformation.py:86

        self,
        non_default_params: dict | None,
        model: str,
        drop_params: bool,
        query: str,
        documents: list[str | dict[str, Any]],
        custom_llm_provider: str | None = None,
        top_n: int | None = None,
        rank_fields: list[str] | None = None,
        return_documents: bool | None = True,
        max_chunks_per_doc: int | None = None,
        max_tokens_per_doc: int | None = None,
        instruction: str | None = None,
    ) -> dict:
        """
        Map parameters for Hosted VLLM rerank
        """
        if max_chunks_per_doc is not None:
            raise ValueError("Hosted VLLM does not support max_chunks_per_doc")

        mapped_params: Final = OptionalRerankParams(
            query=query,
            documents=documents,
            top_n=top_n,
            rank_fields=rank_fields,
            return_documents=return_documents,
        )

        # `instruction` is a vLLM-supported passthrough (folded into the model's
        # chat_template_kwargs). Only forward it when explicitly set so omitting
        # it leaves the request unchanged.
        if instruction is not None:
            mapped_params["instruction"] = instruction

        return dict(mapped_params)

    def validate_environment(

View on GitHub (pinned to 77b7c6c40c)