BerriAI/litellm · error · ValueError

Error parsing response: {raw_response.text}, status_code={ra

Error message

Error parsing response: {raw_response.text}, status_code={raw_response.status_code}

What it means

Raised in HostedVLLM RerankConfig.transform_rerank_response when raw_response.json() throws — i.e. the vLLM rerank endpoint returned a body that is not valid JSON. The ValueError embeds the raw text and HTTP status code so you can see what the server actually returned (often an HTML error page, empty body, or a gateway error).

Source

Thrown at litellm/llms/hosted_vllm/rerank/transformation.py:167

    def transform_rerank_response(
        self,
        model: str,
        raw_response: httpx.Response,
        model_response: RerankResponse,
        logging_obj: LiteLLMLoggingObj,
        api_key: str | None = None,
        request_data: dict = {},
        optional_params: dict = {},
        litellm_params: dict = {},
    ) -> RerankResponse:
        """
        Process response from Hosted VLLM rerank API
        """
        try:
            raw_response_json: Final = raw_response.json()
        except Exception:
            raise ValueError(f"Error parsing response: {raw_response.text}, status_code={raw_response.status_code}")

        return self._transform_response(raw_response_json)

    def get_error_class(self, error_message: str, status_code: int, headers: dict | httpx.Headers) -> BaseLLMException:
        return HostedVLLMRerankError(message=error_message, status_code=status_code, headers=headers)

    def _transform_response(self, response: dict) -> RerankResponse:
        # Extract usage information
        usage_data: Final = response.get("usage", {})
        _billed_units: Final = RerankBilledUnits(total_tokens=usage_data.get("total_tokens", 0))
        _tokens: Final = RerankTokens(input_tokens=usage_data.get("total_tokens", 0))
        rerank_meta: Final = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens)

        # Extract results
        _results: Final[list[dict] | None] = response.get("results")

        if _results is None:
            raise ValueError(f"No results found in the response={response}")

View on GitHub (pinned to 6c2dcb801b)

Solutions

Read the embedded status_code and body text in the message: 502/503 → proxy/upstream issue, fix or restart the vLLM service; 404 → wrong api_base path; empty body → connection-level failure.
Verify the endpoint manually: curl -X POST <api_base>/rerank -H 'Content-Type: application/json' -d '{"model":"...","query":"q","documents":["a"]}' and inspect the raw response.
Check api_base points at the vLLM server (and correct port), not a UI or proxy route.
Add retry with backoff at the caller for transient gateway failures.

Example fix

# before
result = litellm.rerank(model='hosted_vllm/reranker', query=q, documents=docs,
                        api_base='http://vllm:8000')
# ValueError: Error parsing response: <html>502 Bad Gateway</html>, status_code=502

# after
import time
for attempt in range(3):
    try:
        result = litellm.rerank(model='hosted_vllm/reranker', query=q,
                                documents=docs, api_base='http://vllm:8000')
        break
    except ValueError as e:
        if attempt == 2:
            raise
        time.sleep(2 ** attempt)

Defensive patterns

Strategy: retry

Validate before calling

null  # cannot fully prevent: server/proxy behavior; but can preflight the URL

import httpx

def preflight_rerank_endpoint(api_base: str) -> None:
    r = httpx.get(api_base.rstrip("/") + "/health", timeout=5)
    r.raise_for_status()

Try / catch

try:
    result = litellm.rerank(model="hosted_vllm/...", query=q, documents=docs, api_base=base)
except ValueError as e:
    if "Error parsing response" in str(e):
        # body/status embedded in message — inspect and retry transient 5xx/gateway errors
        log.warning("vLLM rerank non-JSON response: %s", e)
        backoff_and_retry()
    raise

Prevention

Health-check the vLLM endpoint (e.g. /health) before request batches.
Keep proxies/load balancers in front of vLLM from returning HTML error pages — return 503 JSON or fail the connection instead.

When it happens

Trigger: vLLM returns non-JSON: 502/503 HTML from a proxy or gateway in front of vLLM, connection reset producing an empty body, wrong URL hitting a non-API route (server returns plain text/404 page), or the server crashing mid-request.

Common situations: Reverse proxy (nginx/traefik/k8s ingress) in front of vLLM returning its own error page; api_base accidentally pointing at the wrong port or path so the response is not the rerank API; vLLM pod restarted/OOM-killed during the request; mixed http/https or auth redirect returning HTML.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/3db7cd7d0da576b1. Report an issue: GitHub.