BerriAI/litellm · error · BaseLLMException

{raw_response.text}

Error message

{raw_response.text}

What it means

Raised by litellm's Nvidia NIM rerank response transformer when raw_response.json() fails: the NIM /v1/ranking endpoint returned a non-JSON body. The exception message is the raw response text, and the upstream status code and headers are propagated on the BaseLLMException.

Source

Thrown at litellm/llms/nvidia_nim/rerank/transformation.py:296

                }
            ]
        }

        LiteLLM expects (RerankResponse):
        {
            "results": [
                {
                    "index": 0,
                    "relevance_score": 0.123,
                    "document": {"text": "..."}  # optional
                }
            ]
        }
        """
        try:
            raw_response_json: Final = raw_response.json()
        except Exception:
            raise BaseLLMException(
                status_code=raw_response.status_code,
                message=raw_response.text,
                headers=raw_response.headers,
            )

        # Parse as NvidiaNimRerankResponse
        nvidia_response: Final[NvidiaNimRerankResponse] = raw_response_json

        # Transform Nvidia NIM response to LiteLLM format
        results: Final[list[RerankResponseResult]] = []
        rankings: Final = nvidia_response.get("rankings", [])

        # Get original documents from request if we need to include them
        original_passages: Final[list[NvidiaNimPassageObject]] = request_data.get("passages", [])

        for ranking in rankings:
            result_item: RerankResponseResult = {
                "index": ranking["index"],

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Read the message/status_code — the literal upstream body identifies auth vs routing vs outage.
2. Confirm api_base is the NIM ranking route (e.g. http://host:8000/v1/ranking or the hosted equivalent).
3. For self-hosted NIM, verify the container is up and the reranking model is loaded (curl the /v1/models endpoint).
4. Refresh NVIDIA_NIM_API_KEY if targeting the hosted endpoint.

Example fix

# before
litellm.rerank(model="nvidia_nim/nv-rerankqa-mistral-4b-v3", query=q, documents=docs)  # wrong base

# after
litellm.rerank(
    model="nvidia_nim/nv-rerankqa-mistral-4b-v3",
    query=q,
    documents=docs,
    api_base="http://nim-host:8000/v1/ranking",
    api_key=os.environ["NVIDIA_NIM_API_KEY"],
)

import httpx, os
base = "http://nim-host:8000"
resp = httpx.get(f"{base}/v1/models", timeout=5)  # preflight: is the NIM up?
assert resp.status_code == 200, f"NIM unreachable at {base}"

from litellm.exceptions import APIError
try:
    res = litellm.rerank(model="nvidia_nim/nv-rerankqa-mistral-4b-v3", query=q, documents=docs)
except APIError as e:
    status = getattr(e, "status_code", None)
    if status == 401:
        raise RuntimeError("Invalid NVIDIA_NIM_API_KEY") from e
    if status and status >= 500:
        return retry_with_backoff()  # gateway/nim down
    raise

When it happens

Trigger: Reranking against nvidia_nim/* when the endpoint returns HTML/plain text: invalid NVIDIA_NIM_API_KEY (401 HTML), wrong api_base pointing at a UI page, a self-hosted NIM container that is down (502 from ingress), or model not deployed on the local NIM.

Common situations: Wrong api_base for self-hosted NIM (missing /v1 suffix or pointing at the docs UI), expired nvapi key against build.nvidia.com, or ingress/gateway errors in Kubernetes.

Related errors

• api_base must be provided for Hosted VLLM rerank
• api_base is required for Infinity rerank
• top_n must be a positive integer, got: {top_n!r}
• Nvidia NIM API key is required. Please set 'NVIDIA_NIM_API_K
• query is required for Nvidia NIM rerank

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/ae727c0b9155c1ca. Report an issue: GitHub.