BerriAI/litellm · error · TritonError

raw_response.text

Error message

raw_response.text

What it means

This is the Triton /generate response handler: it calls raw_response.json() and, if parsing fails, raises TritonError carrying raw_response.text and the HTTP status code. A JSON decode failure means Triton answered with a non-JSON body — typically an error page (model not loaded, bad input shape, 400/500 HTML) — so the original error text is preserved in the exception message.

Source

Thrown at litellm/llms/triton/completion/transformation.py:222

    def transform_response(
        self,
        model: str,
        raw_response: Response,
        model_response: ModelResponse,
        logging_obj: LiteLLMLoggingObj,
        request_data: dict,
        messages: list[AllMessageValues],
        optional_params: dict,
        litellm_params: dict,
        encoding: Any,
        api_key: str | None = None,
        json_mode: bool | None = None,
    ) -> ModelResponse:
        try:
            raw_response_json: Final = raw_response.json()
        except Exception:
            raise TritonError(message=raw_response.text, status_code=raw_response.status_code)
        model_response.choices = [Choices(index=0, message=Message(content=raw_response_json["text_output"]))]

        return model_response


class TritonInferConfig(TritonConfig):
    """
    Transformations for triton /infer endpoint (his is an infer model with a custom model on triton)
    """

    def transform_request(
        self,
        model: str,
        messages: list[AllMessageValues],
        optional_params: dict,
        litellm_params: dict,
        headers: dict,
    ) -> dict:

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Read the TritonError message — it contains the raw body (often 'Request for unknown model' or shape errors).
2. Confirm api_base ends in /generate for tensorrt-llm/fastertransformer style backends and that model names match Triton's loaded models.
3. Check the model loads cleanly in Triton (curl the model ready endpoint /v2/models/<m>/ready).
4. Fix input config: ensure the model's expected inputs (text_input, ..., parameters) align with the request LiteLLM builds.

Example fix

# before
resp = litellm.completion(
    model="triton/my-llm",
    messages=msgs,
    api_base="http://triton:8000/v2/models/my-llm/infer",  # wrong suffix for this handler
)
# TritonError: [500] ... non-JSON body ...

# after
resp = litellm.completion(
    model="triton/my-llm",
    messages=msgs,
    api_base="http://triton:8000/v2/models/my-llm/generate",
)

import httpx

def triton_model_ready(host: str, model: str) -> bool:
    """Cheap readiness probe before routing generate traffic."""
    try:
        return (
            httpx.get(f"{host}/v2/models/{model}/ready", timeout=2).status_code == 200
        )
    except httpx.HTTPError:
        return False

from litellm.llms.triton.common_utils import TritonError  # if exposed; else catch via attribute

try:
    resp = litellm.completion(model="triton/my-llm", messages=msgs, api_base=base)
except Exception as e:
    # TritonError carries Triton's raw body + status — always surface it
    if getattr(e, "status_code", None) is not None:
        logger.error("triton generate failed [%s]: %s", e.status_code, e)
        raise
    raise

When it happens

Trigger: Triton returns 400 because the generate request payload doesn't match the model config (wrong input names/shapes); the model failed to load (404 'model not found'); server errors return HTML/plain-text; pointing api_base at /infer while using the generate parser so the body shape mismatches.

Common situations: Model config.json input tensor names not matching what LiteLLM sends; Triton model still loading after deploy (503 text); version mismatch between Triton's response envelope and the endpoint type; reverse proxies (nginx) intercepting with HTML error pages.

Related errors

• raw_response.text
• Failed to decode JSON from chunk: {chunk}
• api_base is required
• Invalid Triton API base: {api_base}
• api_base is required for triton. Please pass `api_base`

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/14957ee716e25d56. Report an issue: GitHub.