BerriAI/litellm · error · ValueError

GET eval is not supported for {custom_llm_provider}

Error message

GET eval is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.get_eval (litellm/evals/main.py:514) when the provider passed to the call has no Evals API config in ProviderConfigManager. Only LlmProviders.OPENAI maps to a config (OpenAIEvalsConfig); all other providers return None and the GET-by-eval-id call is refused before validate_environment/URL building.

Source

Thrown at litellm/evals/main.py:514

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("aget_eval", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"GET eval is not supported for {custom_llm_provider}")

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

        # Transform request
        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE
        url, headers = evals_api_provider_config.transform_get_eval_request(
            eval_id=eval_id,
            api_base=api_base,
            litellm_params=litellm_params,
            headers=headers,
        )

        # Pre-call logging
        litellm_logging_obj.update_from_kwargs(
            kwargs=kwargs,
            model=None,

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Call with custom_llm_provider='openai' or omit it (defaults to 'openai')
2. Confirm OPENAI_API_KEY is set since the OpenAI config's validate_environment will require it next
3. For other providers, hit their eval endpoints directly rather than through litellm.evals

Example fix

# before
e = litellm.evals.get_eval(eval_id="eval_123", custom_llm_provider="azure")

# after
e = litellm.evals.get_eval(eval_id="eval_123", custom_llm_provider="openai")

if custom_llm_provider != "openai":
    raise ValueError("get_eval requires the OpenAI provider")
litellm.evals.get_eval(eval_id=eval_id, custom_llm_provider="openai")

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

try:
    litellm.evals.get_eval(eval_id=eval_id)
except ValueError as e:
    if "GET eval is not supported" in str(e):
        logger.error("evals are OpenAI-only in litellm")
        raise

When it happens

Trigger: Calling litellm.evals.get_eval(eval_id=..., custom_llm_provider='azure'/'anthropic'/etc.); retrieving an eval created on OpenAI but with a provider string left over from other litellm calls.

Common situations: Hybrid apps that reuse a single provider constant across completion, files, and evals helpers; migrating evals off OpenAI and expecting LiteLLM parity.

Related errors

• CREATE eval is not supported for {custom_llm_provider}
• LIST evals is not supported for {custom_llm_provider}
• UPDATE eval is not supported for {custom_llm_provider}
• DELETE eval is not supported for {custom_llm_provider}
• CANCEL eval is not supported for {custom_llm_provider}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/408223eb3dee64b6. Report an issue: GitHub.