BerriAI/litellm · error · ValueError

LIST evals is not supported for {custom_llm_provider}

Error message

LIST evals is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.list_evals (litellm/evals/main.py:348) when the resolved provider has no Evals API config. ProviderConfigManager.get_provider_evals_api_config only returns a config for LlmProviders.OPENAI; every other provider yields None and triggers this error before any list parameters are applied.

Source

Thrown at litellm/evals/main.py:348

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("alist_evals", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"LIST evals is not supported for {custom_llm_provider}")

        # Build list parameters
        list_params: Final[ListEvalsParams] = {}
        if limit is not None:
            list_params["limit"] = limit
        if after is not None:
            list_params["after"] = after
        if before is not None:
            list_params["before"] = before
        if order is not None:
            list_params["order"] = order
        if order_by is not None:
            list_params["order_by"] = order_by

        # Merge extra_query if provided
        if extra_query:
            list_params.update(extra_query)

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Use custom_llm_provider='openai' (or omit the argument) so the OpenAI Evals config is selected
2. Query the target provider's eval API directly via HTTP if you need non-OpenAI eval listings
3. Check litellm/utils.py get_provider_evals_api_config for the currently supported provider list before upgrading assumptions

Example fix

# before
evals = litellm.evals.list_evals(custom_llm_provider="azure")

# after
evals = litellm.evals.list_evals(custom_llm_provider="openai")

if custom_llm_provider != "openai":
    raise ValueError("listing evals requires custom_llm_provider='openai'")
litellm.evals.list_evals(custom_llm_provider="openai", ...)

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

try:
    litellm.evals.list_evals(...)
except ValueError as e:
    if "LIST evals is not supported" in str(e):
        return []  # or surface 'unsupported provider' to the UI

When it happens

Trigger: Calling litellm.evals.list_evals(custom_llm_provider='anthropic'|'azure'|'bedrock'|...) or any non-'openai' value; omitting the provider is safe because it defaults to 'openai'.

Common situations: Switching an eval-dashboard integration from OpenAI to another provider while keeping litellm.evals calls; copy-pasted provider strings from completion() code into evals calls.

Related errors

• CREATE eval is not supported for {custom_llm_provider}
• GET eval is not supported for {custom_llm_provider}
• UPDATE eval is not supported for {custom_llm_provider}
• DELETE eval is not supported for {custom_llm_provider}
• CANCEL eval is not supported for {custom_llm_provider}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/0bc6cc0a3d0ea4ef. Report an issue: GitHub.