BerriAI/litellm · error · InfinityError

{raw_response.text}

Error message

{raw_response.text}

What it means

Raised when the Infinity rerank response body fails JSON parsing. The raw text and status are forwarded as InfinityError, indicating the request reached something that does not return JSON — wrong route, down service, or an intermediary error page.

Source

Thrown at litellm/llms/infinity/rerank/transformation.py:85

        self,
        model: str,
        raw_response: httpx.Response,
        model_response: RerankResponse,
        logging_obj: LiteLLMLoggingObj,
        api_key: str | None = None,
        request_data: dict = {},
        optional_params: dict = {},
        litellm_params: dict = {},
    ) -> RerankResponse:
        """
        Transform Infinity rerank response

        No transformation required, Infinity follows Cohere API response format
        """
        try:
            raw_response_json: Final = raw_response.json()
        except Exception:
            raise InfinityError(message=raw_response.text, status_code=raw_response.status_code)

        _billed_units: Final = RerankBilledUnits(**raw_response_json.get("usage", {}))
        _tokens: Final = RerankTokens(
            input_tokens=raw_response_json.get("usage", {}).get("prompt_tokens", 0),
            output_tokens=(
                raw_response_json.get("usage", {}).get("total_tokens", 0)
                - raw_response_json.get("usage", {}).get("prompt_tokens", 0)
            ),
        )
        rerank_meta: Final = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens)

        cohere_results: Final[list[RerankResponseResult]] = []
        if raw_response_json.get("results"):
            for result in raw_response_json.get("results"):
                _rerank_response = RerankResponseResult(
                    index=result.get("index"),
                    relevance_score=result.get("relevance_score"),
                )

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Read the forwarded body text to identify what answered.
2. Confirm the Infinity version supports /rerank: curl -X POST http://<base>/rerank.
3. Fix or pin the deployment so the rerank route exists at api_base.

def rerank_route_exists(base: str, api_key: str | None = None) -> bool:
    import httpx
    r = httpx.post(f'{base.rstrip("/")}/rerank', json={'query': 'x', 'texts': ['y']},
                   headers={'Authorization': f'Bearer {api_key}'} if api_key else {},
                   timeout=5)
    return r.status_code != 404 and 'json' in r.headers.get('content-type', '')

try:
    out = litellm.rerank(model='infinity/r', query=q, documents=docs, api_base=base)
except Exception as e:
    if type(e).__name__ == 'InfinityError' and '404' not in str(e):
        raise
    raise RuntimeError(f'Infinity at {base} lacks /rerank or returned non-JSON') from e

When it happens

Trigger: api_base for rerank points at a server/route that replies with HTML or empty body; Infinity version that does not implement /rerank returns a plain-text 404; service restarted mid-request.

Common situations: Older Infinity versions without a rerank endpoint; nginx default pages; wrong path prefix in a containerized deployment.

Related errors

• {raw_response.text}
• {raw_response.text}
• api_base is required for Infinity rerank
• No results found in the response={raw_response_json}
• Original Response received: {raw_response.text}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/b0156d4f5f7ade3d. Report an issue: GitHub.