BerriAI/litellm · error · ValueError

UPDATE eval is not supported for {custom_llm_provider}

Error message

UPDATE eval is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.update_eval (litellm/evals/main.py:679) when the resolved provider lacks an Evals API config. ProviderConfigManager.get_provider_evals_api_config supports only LlmProviders.OPENAI, so update operations against any other provider are rejected before the update_request dict is built.

Source

Thrown at litellm/evals/main.py:679

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("aupdate_eval", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"UPDATE eval is not supported for {custom_llm_provider}")

        # Build update request
        update_request: Final[UpdateEvalRequest] = {}
        if name is not None:
            update_request["name"] = name

        # Filter metadata to exclude internal LiteLLM fields
        if metadata is not None:
            # List of internal LiteLLM metadata keys that should NOT be sent to OpenAI
            internal_keys: Final = {
                "headers",
                "requester_metadata",
                "user_api_key_hash",
                "user_api_key_alias",
                "user_api_key_spend",
                "user_api_key_max_budget",
                "user_api_key_team_id",
                "user_api_key_user_id",

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Pass custom_llm_provider='openai' (default) with a valid OPENAI_API_KEY
2. Use the provider's native REST API for eval updates outside OpenAI
3. Guard eval-management features in your app to OpenAI-only until LiteLLM adds configs

Example fix

# before
litellm.evals.update_eval(eval_id="eval_123", name="v2", custom_llm_provider="anthropic")

# after
litellm.evals.update_eval(eval_id="eval_123", name="v2", custom_llm_provider="openai")

if custom_llm_provider != "openai":
    raise ValueError("update_eval requires the OpenAI provider")
litellm.evals.update_eval(eval_id=eval_id, custom_llm_provider="openai", ...)

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

try:
    litellm.evals.update_eval(eval_id=eval_id, name=name)
except ValueError as e:
    if "UPDATE eval is not supported" in str(e):
        logger.error("switch evals calls to custom_llm_provider='openai'")
        raise

When it happens

Trigger: Calling litellm.evals.update_eval(eval_id=..., custom_llm_provider=<non-openai>); attempting to rename or update testing criteria on an eval while pointed at Azure/Anthropic/Bedrock.

Common situations: Multi-provider deployments reusing a provider variable; assuming evals CRUD works everywhere because completion() does.

Related errors

• CREATE eval is not supported for {custom_llm_provider}
• LIST evals is not supported for {custom_llm_provider}
• GET eval is not supported for {custom_llm_provider}
• DELETE eval is not supported for {custom_llm_provider}
• CANCEL eval is not supported for {custom_llm_provider}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/8ac48f7bea8b55d3. Report an issue: GitHub.