BerriAI/litellm · error · ValueError

api_base is required for triton. Please pass `api_base`

Error message

api_base is required for triton. Please pass `api_base`

What it means

Triton embeddings are served by a self-hosted Triton Inference Server, so unlike cloud providers there is no default URL to fall back to. litellm hard-requires the server address via api_base and raises immediately when it is None.

Source

Thrown at litellm/main.py:6488

            response = bedrock_embedding.embeddings(
                model=model,
                input=transformed_input,
                encoding=_get_encoding(),
                logging_obj=logging,
                optional_params=optional_params,
                model_response=EmbeddingResponse(),
                client=client,
                timeout=timeout,
                aembedding=aembedding,
                litellm_params={},
                api_base=api_base,
                print_verbose=print_verbose,
                extra_headers=headers,
                api_key=api_key,
            )
        elif custom_llm_provider == "triton":
            if api_base is None:
                raise ValueError("api_base is required for triton. Please pass `api_base`")
            response = base_llm_http_handler.embedding(
                model=model,
                input=input,
                custom_llm_provider=custom_llm_provider,
                api_base=api_base,
                api_key=api_key,
                logging_obj=logging,
                timeout=timeout,
                model_response=EmbeddingResponse(),
                optional_params=optional_params,
                client=client,
                aembedding=aembedding,
                litellm_params={},
            )
        elif custom_llm_provider == "gemini":
            gemini_api_key: Final = api_key or get_api_key_from_env() or litellm.api_key

            api_base = api_base or litellm.api_base or get_secret_str("GEMINI_API_BASE")

View on GitHub (pinned to 77b7c6c40c)

Solutions

  1. Pass the Triton HTTP endpoint explicitly: litellm.embedding(model='triton/E5', input=[...], api_base='http://triton-host:8000')
  2. Or set litellm.api_base for the whole process if all calls hit the same server
  3. Verify the server is up: curl http://triton-host:8000/v2/health/ready

Example fix

# before
resp = litellm.embedding(model="triton/E5", input=["hi"])

# after
resp = litellm.embedding(model="triton/E5", input=["hi"], api_base="http://triton-host:8000")
Defensive patterns

Strategy: validation

Validate before calling

TRITON_URL = "http://triton-host:8000"
if not TRITON_URL:
    raise SystemExit("triton api_base missing")
resp = litellm.embedding(model="triton/E5", input=["hi"], api_base=TRITON_URL)

Try / catch

try:
    resp = litellm.embedding(model="triton/E5", input=["hi"], api_base=TRITON_URL)
except ValueError as e:
    if "api_base is required for triton" in str(e):
        raise RuntimeError("Triton endpoint not configured") from e

Prevention

When it happens

Trigger: litellm.embedding(model='triton/<served-model>', input=[...]) without an api_base kwarg and without litellm.api_base set.

Common situations: Copying Triton examples that omit the endpoint; the Triton pod's DNS name or port changed after redeploy; assuming an env var like TRITON_API_BASE is read automatically when only the kwarg is checked.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/7cc516ea5bb2ef28. Report an issue: GitHub.