BerriAI/litellm · error · HuggingFaceError

{raw_response.text}

Error message

{raw_response.text}

What it means

Raised when the HuggingFace rerank endpoint's response body cannot be parsed as JSON. The message is the raw body text (or str(response) as fallback) and the status defaults to 500 when unavailable — typically an HTML error page or empty body from a gateway/proxy in front of the rerank service.

Source

Thrown at litellm/llms/huggingface/rerank/transformation.py:181

        request_body.update(optional_rerank_params)

        return request_body

    def transform_rerank_response(
        self,
        model: str,
        raw_response: httpx.Response,
        model_response: RerankResponse,
        logging_obj: LoggingClass,
        api_key: str | None = None,
        request_data: dict = {},
        optional_params: dict = {},
        litellm_params: dict = {},
    ) -> RerankResponse:
        try:
            raw_response_json: Final[HuggingFaceRerankResponseList] = raw_response.json()
        except Exception:
            raise HuggingFaceError(
                message=getattr(raw_response, "text", str(raw_response)),
                status_code=getattr(raw_response, "status_code", 500),
            )

        # Use standard litellm token counter for proper token estimation
        input_text = request_data.get("query", "")
        try:
            # Calculate tokens for the raw response JSON string
            response_text: Final = str(raw_response_json)
            estimated_output_tokens = token_counter(model=model, text=response_text)

            # Calculate input tokens from query and documents
            query: Final = request_data.get("query", "")
            documents: Final = request_data.get("texts", [])

            # Convert documents to string if they're not already
            documents_text = ""
            for doc in documents:

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Inspect the raw body in the message to identify the proxy/gateway source.
2. Fix api_base to the actual rerank route; confirm with curl.
3. Add retry-with-backoff for transient 5xx gateway responses.

def endpoint_serves_json(url: str) -> bool:
    import httpx
    try:
        return 'json' in httpx.get(url, timeout=5).headers.get('content-type', '')
    except Exception:
        return False

try:
    result = litellm.rerank(model=model, query=q, documents=docs, api_base=base)
except litellm.llms.huggingface.common_utils.HuggingFaceError as e:
    if '<html' in str(e).lower():
        raise RuntimeError(f'{base} is fronted by an HTML error page; check deployment') from e
    raise

When it happens

Trigger: Self-hosted reranker (e.g. Infinity/text-embeddings-inference) behind a reverse proxy returning HTML 502, wrong api_base path returning the web UI, or a crashed worker returning an empty body.

Common situations: Kubernetes ingress/nginx returning error pages during rollout, or a port/path mismatch so the request never reaches the rerank handler.

Related errors

• Original Response received: {raw_response.text}
• {raw_response.text}
• Error parsing response: {raw_response.text}, status_code={ra
• reranker requires 2+ sentences
• response is not in expected format - {completion_response}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/5ea8767ff9793fff. Report an issue: GitHub.