BerriAI/litellm · error · ValueError

Invalid Triton API base: {api_base}

Error message

Invalid Triton API base: {api_base}

What it means

LiteLLM's Triton integration decides how to build/parse requests by inspecting the api_base suffix: URLs ending in /generate use the 'generate' schema, those ending in /infer use the custom-model 'infer' schema. Any api_base not ending with one of those two suffixes raises this ValueError with the offending URL. It is a client-side URL contract check, not a network call.

Source

Thrown at litellm/llms/triton/completion/transformation.py:165

                headers=headers,
            )
        elif llm_type == "infer":
            return TritonInferConfig().transform_request(
                model=model,
                messages=messages,
                optional_params=optional_params,
                litellm_params=litellm_params,
                headers=headers,
            )
        return {}

    def _get_triton_llm_type(self, api_base: str) -> Literal["generate", "infer"]:
        if api_base.endswith("/generate"):
            return "generate"
        elif api_base.endswith("/infer"):
            return "infer"
        else:
            raise ValueError(f"Invalid Triton API base: {api_base}")

    def get_model_response_iterator(
        self,
        streaming_response: Iterator[str] | AsyncIterator[str] | ModelResponse,
        sync_stream: bool,
        json_mode: bool | None = False,
    ) -> Any:
        return TritonResponseIterator(
            streaming_response=streaming_response,
            sync_stream=sync_stream,
            json_mode=json_mode,
        )


class TritonGenerateConfig(TritonConfig):
    """
    Transformations for triton /generate endpoint (This is a trtllm model)
    """

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Use the full endpoint ending in /generate: http://<host>:8000/v2/models/<model>/generate.
2. Or end it in /infer for custom Python/backend models: http://<host>:8000/v2/models/<model>/infer.
3. Remove trailing slashes or query fragments after the suffix.
4. Target the HTTP port (default 8000), not gRPC 8001 or metrics 8002.

Example fix

# before
resp = litellm.completion(
    model="triton/my-llm",
    messages=msgs,
    api_base="http://triton:8000/v2/models/my-llm",   # no /generate suffix
)
# -> ValueError: Invalid Triton API base: http://triton:8000/v2/models/my-llm

# after
resp = litellm.completion(
    model="triton/my-llm",
    messages=msgs,
    api_base="http://triton:8000/v2/models/my-llm/generate",
)

def valid_triton_api_base(url: str) -> bool:
    """Triton endpoints must end with /generate or /infer."""
    return url.endswith("/generate") or url.endswith("/infer")


assert valid_triton_api_base(cfg["api_base"]), (
    "api_base must be http://<host>:8000/v2/models/<model>/{generate|infer}"
)

from typing import TypeGuard

def is_triton_generate_url(url: str) -> TypeGuard[str]:
    """Narrows to URLs the generate handler can use."""
    return url.endswith("/generate")

def is_triton_infer_url(url: str) -> TypeGuard[str]:
    """Narrows to URLs the infer handler can use."""
    return url.endswith("/infer")

try:
    resp = litellm.completion(model="triton/my-llm", messages=msgs, api_base=base)
except ValueError as e:
    if "Invalid Triton API base" in str(e):
        base = base.rstrip("/") + ("/generate" if backend == "generate" else "/infer")
        resp = litellm.completion(model="triton/my-llm", messages=msgs, api_base=base)
    else:
        raise

When it happens

Trigger: Passing api_base="http://triton:8000" (server root), "http://triton:8000/v2/models/my-llm" (missing the method), or a URL with a trailing slash or query string after generate/infer; copying the gRPC (8001) or metrics (8002) port instead of HTTP 8000.

Common situations: First-time Triton users pasting the server base URL instead of the full model endpoint; switching a model from a generate backend to an ensemble/infer backend and forgetting to update the suffix; proxies that normalize/strip URL parts; trailing-slash additions by config tooling.

Related errors

• api_base is required
• raw_response.text
• raw_response.text
• api_base is required for triton. Please pass `api_base`
• Compresr guardrail api_base must be http or https, got schem

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/7bbd44d5a6a707a2. Report an issue: GitHub.