BerriAI/litellm · error · ValueError

VLLM_API_BASE or VLLM_API_KEY is not set. Please set the env

Error message

VLLM_API_BASE or VLLM_API_KEY is not set. Please set the environment variable, to query VLLM's `/models` endpoint.

What it means

Raised by VLLMModelInfo.get_models when api_base or api_key is None after resolution. Critical detail: on this code path get_api_key() always returns None (litellm/llms/vllm/common_utils.py:56), so the check can never pass — get_models() for vLLM raises this ValueError unconditionally, even when VLLM_API_BASE and VLLM_API_KEY are both set. It is a library defect, not purely a configuration problem.

Source

Thrown at litellm/llms/vllm/common_utils.py:68

            raise ValueError(
                "VLLM_API_BASE is not set. Please set the environment variable, to use VLLM's pass-through - `{LITELLM_API_BASE}/vllm/{endpoint}`."
            )
        return api_base

    @staticmethod
    def get_api_key(api_key: str | None = None) -> str | None:
        return None

    @staticmethod
    def get_base_model(model: str) -> str | None:
        return model

    def get_models(self, api_key: str | None = None, api_base: str | None = None) -> list[str]:
        api_base = VLLMModelInfo.get_api_base(api_base)
        api_key = VLLMModelInfo.get_api_key(api_key)
        endpoint: Final = "/v1/models"
        if api_base is None or api_key is None:
            raise ValueError(
                "VLLM_API_BASE or VLLM_API_KEY is not set. Please set the environment variable, to query VLLM's `/models` endpoint."
            )

        url: Final = _add_path_to_api_base(api_base, endpoint)
        response: Final = litellm.module_level_client.get(
            url=url,
        )

        response.raise_for_status()

        models: Final = response.json()["data"]

        return [model["id"] for model in models]

    def get_error_class(self, error_message: str, status_code: int, headers: dict | httpx.Headers) -> BaseLLMException:
        return VLLMError(status_code=status_code, message=error_message, headers=headers)

View on GitHub (pinned to 77b7c6c40c)

Solutions

Do not rely on model listing for vllm; hardcode/declare the deployed model ids in your proxy config
Query the server directly: curl $VLLM_API_BASE/v1/models (vLLM needs no real API key)
Patch or subclass VLLMModelInfo.get_api_key to return api_key (or 'EMPTY') and pass the class through your integration
Check for a newer litellm release where the vllm get_models path is fixed before working around it

Example fix

# before
models = provider_config.get_models(api_base="http://localhost:8000")
# always raises: VLLM_API_BASE or VLLM_API_KEY is not set

# after (query directly; vLLM serves an OpenAI-compatible list)
import httpx
resp = httpx.get("http://localhost:8000/v1/models")
resp.raise_for_status()
models = [m["id"] for m in resp.json()["data"]]

# or monkeypatch the broken getter
# litellm.llms.vllm.common_utils.VLLMModelInfo.get_api_key = staticmethod(lambda api_key=None: api_key or "EMPTY")

Defensive patterns

Strategy: fallback

Validate before calling

import os, httpx

def list_vllm_models_direct(api_base: str | None = None) -> list[str]:
    base = api_base or os.environ.get("VLLM_API_BASE")
    if not base:
        raise ConfigError("VLLM_API_BASE is not set")
    resp = httpx.get(f"{base.rstrip('/')}/v1/models", timeout=10)
    resp.raise_for_status()
    return [m["id"] for m in resp.json()["data"]]

Try / catch

try:
    models = provider_config.get_models(api_base=base)
except ValueError as e:
    if "VLLM_API_BASE or VLLM_API_KEY" in str(e):
        # known litellm defect: get_api_key() always returns None -> always raises
        models = [m["id"] for m in httpx.get(f"{base}/v1/models").json()["data"]]
    else:
        raise

Prevention

Declare vLLM model ids statically in proxy config instead of discovering them via get_models
Query $VLLM_API_BASE/v1/models directly — vLLM requires no real API key
Pin/track the litellm version and retest model listing after upgrades; check release notes for the vllm get_models fix

When it happens

Trigger: Any call that lists models for the vllm provider — litellm.models() routed to vLLM, proxy model-listing for a vllm deployment, or get_models(api_base=...) — regardless of which environment variables are set.

Common situations: Proxy startup fetching model lists for a vllm provider; dashboards enumerating available models; users setting VLLM_API_KEY and still hitting the error because the getter ignores it.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/50ff9bd14330680a. Report an issue: GitHub.