BerriAI/litellm · error · ValueError

Error parsing response

Error message

Error parsing response: {raw_response.text}, status_code={raw_response.status_code}

What it means

After vLLM answers a rerank request, transform_rerank_response() calls raw_response.json(); if the body is not valid JSON it raises ValueError embedding the raw body text and the HTTP status code for diagnosis. Common non-JSON bodies include HTML error pages from proxies, plain-text gateway errors (502/503), or empty bodies from crashed connections. The status_code in the message is the key diagnostic.

Solutions

1. Read the embedded status_code and body snippet — an HTML 404 page means wrong path, a 502/503 means gateway/upstream trouble
2. Reproduce outside LiteLLM: curl -X POST <api_base>/rerank with the same payload and inspect the raw response
3. Confirm the vLLM server exposes /rerank and was launched with a reranker model (e.g. jina-reranker, bge-reranker)
4. Fix the base URL/proxy so JSON from vLLM reaches the client; retry only on transient 5xx

Example fix

# before
result = litellm.rerank(model="hosted_vllm/jina-reranker", api_base="http://gateway.internal", query=q, documents=docs)

# after — point directly at the vLLM instance serving /rerank
result = litellm.rerank(model="hosted_vllm/jina-reranker", api_base="http://vllm-rerank.internal:8000", query=q, documents=docs)

import httpx

def rerank_endpoint_speaks_json(base: str) -> bool:
    try:
        r = httpx.post(f"{base.rstrip('/')}/rerank", json={"model": "x", "query": "q", "documents": ["d"]}, timeout=5)
        r.json()
        return True
    except ValueError:
        return False

Catch ValueError around litellm.rerank and parse the embedded status code from the message; treat 5xx/gateway snippets in the body as transient (retry with backoff, bounded attempts) and anything else as configuration failure (halt and alert with the raw body excerpt). Never retry blindly — the message tells you which class of failure occurred.

When it happens

Trigger: api_base pointing at a server that answers /rerank with HTML (a reverse proxy 404/502 page, an auth middleware); a gateway timeout returning a plain-text error; vLLM crashing mid-response; hitting a route that is not the rerank endpoint.

Common situations: Base URL misconfigured (missing port, wrong path, pointing at a frontend instead of vLLM); ingress/load-balancer intercepting the request; vLLM not started with a rerank-capable model so an error page is returned; TLS termination returning an HTML challenge.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

• api_base must be provided for Hosted VLLM rerank
• documents is required for Hosted VLLM rerank
• Hosted VLLM does not support max_chunks_per_doc
• Missing required fields in the result=
• No results found in the response=

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/3db7cd7d0da576b1. Report an issue: GitHub.

Appendix: source

Thrown at litellm/llms/hosted_vllm/rerank/transformation.py:167

    def transform_rerank_response(
        self,
        model: str,
        raw_response: httpx.Response,
        model_response: RerankResponse,
        logging_obj: LiteLLMLoggingObj,
        api_key: str | None = None,
        request_data: dict = {},
        optional_params: dict = {},
        litellm_params: dict = {},
    ) -> RerankResponse:
        """
        Process response from Hosted VLLM rerank API
        """
        try:
            raw_response_json: Final = raw_response.json()
        except Exception:
            raise ValueError(f"Error parsing response: {raw_response.text}, status_code={raw_response.status_code}")

        return self._transform_response(raw_response_json)

    def get_error_class(self, error_message: str, status_code: int, headers: dict | httpx.Headers) -> BaseLLMException:
        return HostedVLLMRerankError(message=error_message, status_code=status_code, headers=headers)

    def _transform_response(self, response: dict) -> RerankResponse:
        # Extract usage information
        usage_data: Final = response.get("usage", {})
        _billed_units: Final = RerankBilledUnits(total_tokens=usage_data.get("total_tokens", 0))
        _tokens: Final = RerankTokens(input_tokens=usage_data.get("total_tokens", 0))
        rerank_meta: Final = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens)

        # Extract results
        _results: Final[list[dict] | None] = response.get("results")

        if _results is None:
            raise ValueError(f"No results found in the response={response}")

View on GitHub (pinned to 77b7c6c40c)