What causes "LIST runs is not supported for {custom_llm_provider}" in BerriAI/litellm?

Call list_runs with custom_llm_provider='openai' or omit the argument

BerriAI/litellm · error · ValueError

LIST runs is not supported for {custom_llm_provider}

Error message

LIST runs is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.list_runs (litellm/evals/main.py:1397) when get_provider_evals_api_config returns None for the provider. Listing runs of an eval is implemented only for OpenAI; every other provider string fails here before pagination parameters (limit/after/before/order) are applied.

Source

Thrown at litellm/evals/main.py:1397

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("alist_runs", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"LIST runs is not supported for {custom_llm_provider}")

        # Build list parameters
        list_params: Final[ListRunsParams] = {}
        if limit is not None:
            list_params["limit"] = limit
        if after is not None:
            list_params["after"] = after
        if before is not None:
            list_params["before"] = before
        if order is not None:
            list_params["order"] = order

        # Merge extra_query if provided
        if extra_query:
            list_params.update(extra_query)

        # Validate environment and get headers
        headers = extra_headers or {}

View on GitHub (pinned to 6c2dcb801b)

Solutions

Call list_runs with custom_llm_provider='openai' or omit the argument
Query run history from the provider's own API for non-OpenAI setups
Centralize the evals provider as a constant ('openai') in your codebase to avoid drift

Example fix

# before
runs = litellm.evals.list_runs(eval_id="eval_123", custom_llm_provider="azure")

# after
runs = litellm.evals.list_runs(eval_id="eval_123", custom_llm_provider="openai")

Defensive patterns

Strategy: validation

Validate before calling

if custom_llm_provider != "openai":
    raise ValueError("list_runs requires the OpenAI provider")
runs = litellm.evals.list_runs(eval_id=eval_id, custom_llm_provider="openai")

Type guard

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

Try / catch

try:
    runs = litellm.evals.list_runs(eval_id=eval_id)
except ValueError as e:
    if "LIST runs is not supported" in str(e):
        return []  # unsupported provider: no run data available

Prevention

Omit custom_llm_provider so the OpenAI default applies
In dashboards, gate run-listing views on the provider allowlist
Cache the supported-provider check once per process instead of per request

When it happens

Trigger: Calling litellm.evals.list_runs(eval_id=..., custom_llm_provider=<non-openai>); monitoring dashboards parameterized by deployment provider.

Common situations: Building eval observability UIs with LiteLLM and assuming cross-provider run listings; leftover provider values in shared config objects.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/f9e624865b013039. Report an issue: GitHub.