BerriAI/litellm · error · Exception

response.text

Error message

response.text

What it means

This is the synchronous TogetherAI rerank path: after POSTing to https://api.together.xyz/v1/rerank, any status other than 200 raises a bare Exception whose message is the raw response body (response.text). The underlying causes are server-side rejections — invalid/expired API key (401), unknown model (404/400), malformed query (422), rate limits or Together outages (429/5xx) — surfaced verbatim from Together's error JSON/HTML.

Source

Thrown at litellm/llms/together_ai/rerank/handler.py:62

        request_data_dict: Final = request_data.dict(exclude_none=True)
        if max_chunks_per_doc is not None:
            raise ValueError("TogetherAI does not support max_chunks_per_doc")

        if _is_async:
            return self.async_rerank(request_data_dict, api_key)  # Call async method

        response: Final = client.post(
            "https://api.together.xyz/v1/rerank",
            headers={
                "accept": "application/json",
                "content-type": "application/json",
                "authorization": f"Bearer {api_key}",
            },
            json=request_data_dict,
        )

        if response.status_code != 200:
            raise Exception(response.text)

        _json_response: Final = response.json()

        return TogetherAIRerankConfig()._transform_response(_json_response)

    async def async_rerank(  # New async method
        self,
        request_data_dict: dict[str, Any],
        api_key: str,
    ) -> RerankResponse:
        client: Final = get_async_httpx_client(llm_provider=litellm.LlmProviders.TOGETHER_AI)  # Use async client

        response: Final = await client.post(
            "https://api.together.xyz/v1/rerank",
            headers={
                "accept": "application/json",
                "content-type": "application/json",
                "authorization": f"Bearer {api_key}",

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Read the exception message — it is Together's raw error body, which names the actual cause (auth, model, rate limit).
2. For 401/403: verify the TOGETHERAI_API_KEY used by the call is valid and has rerank access.
3. For 400/404: confirm the model id exists on Together's models page and is a rerank model (e.g. rerank-english-v2.0).
4. For 429/5xx: retry with exponential backoff (litellm retries / Router num_retries) and reduce request rate.

Example fix

# before
resp = litellm.rerank(model="together_ai/rerank-english-v2.0", query=q, documents=docs)
# Exception: {"message":"Invalid API key","type":"invalid_request_error"}

# after
from litellm import Router
router = Router(
    model_list=[{
        "model_name": "together-rerank",
        "litellm_params": {"model": "together_ai/rerank-english-v2.0"},
    }],
    num_retries=3,           # retries 429/5xx
    retry_after=2,
)
resp = router.rerank(model="together-rerank", query=q, documents=docs)

def together_rerank_request_valid(model: str, api_key: str | None, documents: list) -> tuple[bool, str]:
    if not api_key:
        return False, "missing Together API key"
    if not documents:
        return False, "documents must be non-empty"
    if not model.startswith("together_ai/"):
        return False, "model must be a together_ai rerank model"
    return True, ""

import time

for attempt in range(4):
    try:
        resp = litellm.rerank(model="together_ai/rerank-english-v2.0", query=q, documents=docs)
        break
    except Exception as e:
        msg = str(e)
        if "Invalid API key" in msg or "invalid_api_key" in msg:
            raise RuntimeError("fix TOGETHERAI_API_KEY") from e  # not retryable
        if attempt == 3:
            raise
        time.sleep(2 ** attempt)  # retry 429/5xx with backoff

When it happens

Trigger: litellm.rerank(model="together_ai/rerank-english-v2.0", ...) with a wrong or revoked Together API key; referencing a retired rerank model id; exceeding rate limits; Together API returning 500 during an incident. Any non-200 produces this exception with the provider's body as the message.

Common situations: Key rotation that invalidated a stored TOGETHERAI_API_KEY; deprecation of older rerank model versions; bursty retrival workloads hitting 429s; transient Together 502/503s during deploys.

Related errors

• TogetherAI does not support max_chunks_per_doc
• No results found in the response={response}
• Missing required fields in the result={result}
• TogetherAIException - {error_response['error']}
• TogetherAIException - {error_str}

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/b531b1ebd3126cf8. Report an issue: GitHub.