BerriAI/litellm · error · ValueError

TogetherAI does not support max_chunks_per_doc

Error message

TogetherAI does not support max_chunks_per_doc

What it means

LiteLLM's TogetherAI rerank handler builds a RerankRequest and explicitly rejects max_chunks_per_doc before sending anything to https://api.together.xyz/v1/rerank, because Together's rerank API has no such option. If the caller passes max_chunks_per_doc (even as None-adjacent optional config) and it is not None, a ValueError is raised client-side.

Source

Thrown at litellm/llms/together_ai/rerank/handler.py:46

        return_documents: bool | None = True,
        max_chunks_per_doc: int | None = None,
        _is_async: bool | None = False,
    ) -> RerankResponse:
        client: Final = _get_httpx_client()

        request_data: Final = RerankRequest(
            model=model,
            query=query,
            top_n=top_n,
            documents=documents,
            rank_fields=rank_fields,
            return_documents=return_documents,
        )

        # exclude None values from request_data
        request_data_dict: Final = request_data.dict(exclude_none=True)
        if max_chunks_per_doc is not None:
            raise ValueError("TogetherAI does not support max_chunks_per_doc")

        if _is_async:
            return self.async_rerank(request_data_dict, api_key)  # Call async method

        response: Final = client.post(
            "https://api.together.xyz/v1/rerank",
            headers={
                "accept": "application/json",
                "content-type": "application/json",
                "authorization": f"Bearer {api_key}",
            },
            json=request_data_dict,
        )

        if response.status_code != 200:
            raise Exception(response.text)

        _json_response: Final = response.json()

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Remove max_chunks_per_doc from the rerank call when model starts with together_ai/.
2. Branch your wrapper: pass max_chunks_per_doc only for providers that support it (e.g. Cohere).
3. Chunk long documents yourself before calling TogetherAI rerank if you need chunk-level control.

Example fix

# before
resp = litellm.rerank(
    model="together_ai/rerank-english-v2.0",
    query="What is the capital of the US?",
    documents=["Washington, D.C. ...", "Paris is ..."],
    max_chunks_per_doc=16,
)

# after
resp = litellm.rerank(
    model="together_ai/rerank-english-v2.0",
    query="What is the capital of the US?",
    documents=["Washington, D.C. ...", "Paris is ..."],
)

TOGETHER_UNSUPPORTED_RERANK_PARAMS = {"max_chunks_per_doc"}


def build_rerank_kwargs(model: str, **kwargs) -> dict:
    """Strip params the target rerank provider cannot take."""
    if model.startswith("together_ai/"):
        blocked = TOGETHER_UNSUPPORTED_RERANK_PARAMS & set(kwargs)
        if blocked:
            raise ValueError(f"together_ai rerank does not accept: {sorted(blocked)}")
    return kwargs

try:
    resp = litellm.rerank(model="together_ai/rerank-english-v2.0", query=q, documents=docs, **params)
except ValueError as e:
    if "does not support max_chunks_per_doc" in str(e):
        params.pop("max_chunks_per_doc", None)
        resp = litellm.rerank(model="together_ai/rerank-english-v2.0", query=q, documents=docs, **params)
    else:
        raise

When it happens

Trigger: Calling litellm.rerank(model="together_ai/rerank-english-v2.0", query=..., documents=..., max_chunks_per_doc=...) with any non-None value; generic rerank wrappers that forward the full Cohere-style parameter set to every provider.

Common situations: Sharing one rerank call signature across Cohere and TogetherAI — Cohere accepts max_chunks_per_doc, Together does not; copying sample code from a Cohere rerank guide and switching only the model string; config-driven rerank defaults that set the parameter for all deployments.

Related errors

• response.text
• No results found in the response={response}
• Missing required fields in the result={result}
• TogetherAIException - {error_response['error']}
• TogetherAIException - {error_str}

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/cdd1906ae4ceec23. Report an issue: GitHub.