{"record":{"id":"14957ee716e25d56","repo":"BerriAI/litellm","slug":"raw-response-text-14957e","errorCode":null,"errorMessage":"raw_response.text","messagePattern":"raw_response\\.text","errorType":"exception","errorClass":"TritonError","httpStatus":null,"severity":"error","filePath":"litellm/llms/triton/completion/transformation.py","lineNumber":222,"sourceCode":"\n    def transform_response(\n        self,\n        model: str,\n        raw_response: Response,\n        model_response: ModelResponse,\n        logging_obj: LiteLLMLoggingObj,\n        request_data: dict,\n        messages: list[AllMessageValues],\n        optional_params: dict,\n        litellm_params: dict,\n        encoding: Any,\n        api_key: str | None = None,\n        json_mode: bool | None = None,\n    ) -> ModelResponse:\n        try:\n            raw_response_json: Final = raw_response.json()\n        except Exception:\n            raise TritonError(message=raw_response.text, status_code=raw_response.status_code)\n        model_response.choices = [Choices(index=0, message=Message(content=raw_response_json[\"text_output\"]))]\n\n        return model_response\n\n\nclass TritonInferConfig(TritonConfig):\n    \"\"\"\n    Transformations for triton /infer endpoint (his is an infer model with a custom model on triton)\n    \"\"\"\n\n    def transform_request(\n        self,\n        model: str,\n        messages: list[AllMessageValues],\n        optional_params: dict,\n        litellm_params: dict,\n        headers: dict,\n    ) -> dict:","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/llms/triton/completion/transformation.py#L204-L240","documentation":"This is the Triton /generate response handler: it calls raw_response.json() and, if parsing fails, raises TritonError carrying raw_response.text and the HTTP status code. A JSON decode failure means Triton answered with a non-JSON body — typically an error page (model not loaded, bad input shape, 400/500 HTML) — so the original error text is preserved in the exception message.","triggerScenarios":"Triton returns 400 because the generate request payload doesn't match the model config (wrong input names/shapes); the model failed to load (404 'model not found'); server errors return HTML/plain-text; pointing api_base at /infer while using the generate parser so the body shape mismatches.","commonSituations":"Model config.json input tensor names not matching what LiteLLM sends; Triton model still loading after deploy (503 text); version mismatch between Triton's response envelope and the endpoint type; reverse proxies (nginx) intercepting with HTML error pages.","solutions":["Read the TritonError message — it contains the raw body (often 'Request for unknown model' or shape errors).","Confirm api_base ends in /generate for tensorrt-llm/fastertransformer style backends and that model names match Triton's loaded models.","Check the model loads cleanly in Triton (curl the model ready endpoint /v2/models/<m>/ready).","Fix input config: ensure the model's expected inputs (text_input, ..., parameters) align with the request LiteLLM builds."],"exampleFix":"# before\nresp = litellm.completion(\n    model=\"triton/my-llm\",\n    messages=msgs,\n    api_base=\"http://triton:8000/v2/models/my-llm/infer\",  # wrong suffix for this handler\n)\n# TritonError: [500] ... non-JSON body ...\n\n# after\nresp = litellm.completion(\n    model=\"triton/my-llm\",\n    messages=msgs,\n    api_base=\"http://triton:8000/v2/models/my-llm/generate\",\n)","handlingStrategy":"try-catch","validationCode":"import httpx\n\ndef triton_model_ready(host: str, model: str) -> bool:\n    \"\"\"Cheap readiness probe before routing generate traffic.\"\"\"\n    try:\n        return (\n            httpx.get(f\"{host}/v2/models/{model}/ready\", timeout=2).status_code == 200\n        )\n    except httpx.HTTPError:\n        return False","typeGuard":null,"tryCatchPattern":"from litellm.llms.triton.common_utils import TritonError  # if exposed; else catch via attribute\n\ntry:\n    resp = litellm.completion(model=\"triton/my-llm\", messages=msgs, api_base=base)\nexcept Exception as e:\n    # TritonError carries Triton's raw body + status — always surface it\n    if getattr(e, \"status_code\", None) is not None:\n        logger.error(\"triton generate failed [%s]: %s\", e.status_code, e)\n        raise\n    raise","preventionTips":["Probe /v2/models/<model>/ready before sending completion traffic.","Keep generate-type models on /generate URLs so the matching parser runs.","Log TritonError.status_code and message verbatim; the body names the exact tensor/model problem."],"tags":["triton","self-hosted","response-parsing","json-decode","litellm"],"backgroundTag":"invalid-json-response","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}