BerriAI/litellm · error · TritonError

raw_response.text

Error message

raw_response.text

What it means

The Triton embedding handler parses the HTTP response with raw_response.json(); if the body is not JSON it raises TritonError containing raw_response.text and status_code. For embeddings, Triton's /infer endpoint should return {'outputs': [{'shape': [...], 'data': [...]}]}; a non-JSON body means the server errored out (unknown model, wrong input name, load failure) and returned an error page/plain text instead.

Source

Thrown at litellm/llms/triton/embedding/transformation.py:84

                }
            ]
        }

    def transform_embedding_response(
        self,
        model: str,
        raw_response: httpx.Response,
        model_response: EmbeddingResponse,
        logging_obj: LiteLLMLoggingObj,
        api_key: str | None = None,
        request_data: dict = {},
        optional_params: dict = {},
        litellm_params: dict = {},
    ) -> EmbeddingResponse:
        try:
            raw_response_json: Final = raw_response.json()
        except Exception:
            raise TritonError(message=raw_response.text, status_code=raw_response.status_code)

        _embedding_output: Final = []

        _outputs: Final = raw_response_json["outputs"]
        for output in _outputs:
            _shape = output["shape"]
            _data = output["data"]
            _split_output_data = self.split_embedding_by_shape(_data, _shape)

            for idx, embedding in enumerate(_split_output_data):
                _embedding_output.append(
                    {
                        "object": "embedding",
                        "index": idx,
                        "embedding": embedding,
                    }
                )

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Read the TritonError message — it embeds Triton's raw error text (usually names the unknown model or input).
2. Confirm the model is loaded and ready: curl http://<host>:8000/v2/models/<model>/ready.
3. Check api_base is the full /v2/models/<model>/infer path (satisfying the /infer suffix contract).
4. Align config.pbtxt input names/shapes (e.g. 'input_ids'/'INPUT_TEXT') with what the embedding request sends.

Example fix

# before
resp = litellm.embedding(
    model="triton/my-embed",
    input=["hello world"],
    api_base="http://triton:8000/v2/models/my-embed",  # missing /infer
)
# TritonError: [404] ... non-JSON body ...

# after
resp = litellm.embedding(
    model="triton/my-embed",
    input=["hello world"],
    api_base="http://triton:8000/v2/models/my-embed/infer",
)

import httpx

def triton_embedding_endpoint_ok(base: str, model: str) -> bool:
    """Ready probe + URL shape check before embedding calls."""
    if not (base.endswith("/infer") or base.endswith("/generate")):
        return False
    try:
        return httpx.get(f"{base.rsplit('/v2/', 1)[0]}/v2/models/{model}/ready", timeout=2).status_code == 200
    except httpx.HTTPError:
        return False

try:
    resp = litellm.embedding(
        model="triton/my-embed", input=texts,
        api_base="http://triton:8000/v2/models/my-embed/infer",
    )
except Exception as e:
    if getattr(e, "status_code", None) is not None:  # TritonError with raw body
        logger.error("triton embedding failed [%s]: %s", e.status_code, e)
        raise RuntimeError("check triton model readiness/config") from e
    raise

When it happens

Trigger: Calling litellm.embedding(model="triton/...", input=[...]) with api_base at a Triton /infer endpoint where the model is missing/misconfigured (404 'unavailable model'), input tensor names mismatch, or an ingress returns HTML 502; any non-JSON reply triggers this.

Common situations: Embedding models (e.g. tensorrt/onnx encoders) whose config.pbtxt inputs don't match the payload; K8s ingress errors during rollout; using the server root instead of the full /v2/models/<m>/infer path so Triton returns a docs page.

Related errors

• raw_response.text
• Failed to decode JSON from chunk: {chunk}
• api_base is required
• Invalid Triton API base: {api_base}
• api_base is required for triton. Please pass `api_base`

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/ca656d5af820adb8. Report an issue: GitHub.