{"record":{"id":"7bbd44d5a6a707a2","repo":"BerriAI/litellm","slug":"invalid-triton-api-base-api-base","errorCode":null,"errorMessage":"Invalid Triton API base: {api_base}","messagePattern":"Invalid Triton API base: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/llms/triton/completion/transformation.py","lineNumber":165,"sourceCode":"                headers=headers,\n            )\n        elif llm_type == \"infer\":\n            return TritonInferConfig().transform_request(\n                model=model,\n                messages=messages,\n                optional_params=optional_params,\n                litellm_params=litellm_params,\n                headers=headers,\n            )\n        return {}\n\n    def _get_triton_llm_type(self, api_base: str) -> Literal[\"generate\", \"infer\"]:\n        if api_base.endswith(\"/generate\"):\n            return \"generate\"\n        elif api_base.endswith(\"/infer\"):\n            return \"infer\"\n        else:\n            raise ValueError(f\"Invalid Triton API base: {api_base}\")\n\n    def get_model_response_iterator(\n        self,\n        streaming_response: Iterator[str] | AsyncIterator[str] | ModelResponse,\n        sync_stream: bool,\n        json_mode: bool | None = False,\n    ) -> Any:\n        return TritonResponseIterator(\n            streaming_response=streaming_response,\n            sync_stream=sync_stream,\n            json_mode=json_mode,\n        )\n\n\nclass TritonGenerateConfig(TritonConfig):\n    \"\"\"\n    Transformations for triton /generate endpoint (This is a trtllm model)\n    \"\"\"","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/llms/triton/completion/transformation.py#L147-L183","documentation":"LiteLLM's Triton integration decides how to build/parse requests by inspecting the api_base suffix: URLs ending in /generate use the 'generate' schema, those ending in /infer use the custom-model 'infer' schema. Any api_base not ending with one of those two suffixes raises this ValueError with the offending URL. It is a client-side URL contract check, not a network call.","triggerScenarios":"Passing api_base=\"http://triton:8000\" (server root), \"http://triton:8000/v2/models/my-llm\" (missing the method), or a URL with a trailing slash or query string after generate/infer; copying the gRPC (8001) or metrics (8002) port instead of HTTP 8000.","commonSituations":"First-time Triton users pasting the server base URL instead of the full model endpoint; switching a model from a generate backend to an ensemble/infer backend and forgetting to update the suffix; proxies that normalize/strip URL parts; trailing-slash additions by config tooling.","solutions":["Use the full endpoint ending in /generate: http://<host>:8000/v2/models/<model>/generate.","Or end it in /infer for custom Python/backend models: http://<host>:8000/v2/models/<model>/infer.","Remove trailing slashes or query fragments after the suffix.","Target the HTTP port (default 8000), not gRPC 8001 or metrics 8002."],"exampleFix":"# before\nresp = litellm.completion(\n    model=\"triton/my-llm\",\n    messages=msgs,\n    api_base=\"http://triton:8000/v2/models/my-llm\",   # no /generate suffix\n)\n# -> ValueError: Invalid Triton API base: http://triton:8000/v2/models/my-llm\n\n# after\nresp = litellm.completion(\n    model=\"triton/my-llm\",\n    messages=msgs,\n    api_base=\"http://triton:8000/v2/models/my-llm/generate\",\n)","handlingStrategy":"validation","validationCode":"def valid_triton_api_base(url: str) -> bool:\n    \"\"\"Triton endpoints must end with /generate or /infer.\"\"\"\n    return url.endswith(\"/generate\") or url.endswith(\"/infer\")\n\n\nassert valid_triton_api_base(cfg[\"api_base\"]), (\n    \"api_base must be http://<host>:8000/v2/models/<model>/{generate|infer}\"\n)","typeGuard":"from typing import TypeGuard\n\ndef is_triton_generate_url(url: str) -> TypeGuard[str]:\n    \"\"\"Narrows to URLs the generate handler can use.\"\"\"\n    return url.endswith(\"/generate\")\n\ndef is_triton_infer_url(url: str) -> TypeGuard[str]:\n    \"\"\"Narrows to URLs the infer handler can use.\"\"\"\n    return url.endswith(\"/infer\")","tryCatchPattern":"try:\n    resp = litellm.completion(model=\"triton/my-llm\", messages=msgs, api_base=base)\nexcept ValueError as e:\n    if \"Invalid Triton API base\" in str(e):\n        base = base.rstrip(\"/\") + (\"/generate\" if backend == \"generate\" else \"/infer\")\n        resp = litellm.completion(model=\"triton/my-llm\", messages=msgs, api_base=base)\n    else:\n        raise","preventionTips":["Construct Triton URLs from parts: f\"http://{host}:8000/v2/models/{model}/{method}\" where method is generate or infer.","Reject trailing slashes and query strings at config time.","Use HTTP port 8000 only; 8001 (gRPC) and 8002 (metrics) cannot serve these requests."],"tags":["triton","self-hosted","url-validation","configuration","litellm"],"backgroundTag":"invalid-endpoint-url","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}