BerriAI/litellm · error · ValueError

CREATE eval is not supported for {custom_llm_provider}

Error message

CREATE eval is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.create_eval (litellm/evals/main.py:163) when ProviderConfigManager.get_provider_evals_api_config returns None for the resolved provider. LiteLLM's Evals API bindings currently support only OpenAI (see get_provider_evals_api_config in litellm/utils.py, which returns OpenAIEvalsConfig for LlmProviders.OPENAI and None otherwise), so any other custom_llm_provider is rejected before a request is built.

Source

Thrown at litellm/evals/main.py:163

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("acreate_eval", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"CREATE eval is not supported for {custom_llm_provider}")

        # Build create request
        create_request: Final[CreateEvalRequest] = {
            "data_source_config": data_source_config,
            "testing_criteria": testing_criteria,
        }
        if name is not None:
            create_request["name"] = name

        # Merge extra_body if provided
        if extra_body:
            create_request.update(extra_body)

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

        # Transform request

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Pass custom_llm_provider='openai' (or omit it — it defaults to 'openai') and use an OpenAI API key
2. For Azure OpenAI, point api_base at your Azure endpoint with the openai provider if your setup requires it, or call the Azure Evals REST endpoint directly
3. Track LiteLLM release notes for new evals provider configs before assuming support

Example fix

# before
litellm.evals.create_eval(custom_llm_provider="azure", ...)

# after
litellm.evals.create_eval(custom_llm_provider="openai", ...)

import litellm
SUPPORTED_EVALS_PROVIDERS = {litellm.LlmProviders.OPENAI.value}
if custom_llm_provider not in SUPPORTED_EVALS_PROVIDERS:
    raise ValueError("litellm.evals supports only 'openai'")

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

try:
    litellm.evals.create_eval(...)
except ValueError as e:
    if "CREATE eval is not supported" in str(e):
        logger.error("Use custom_llm_provider='openai' for evals")
        raise

When it happens

Trigger: Calling litellm.evals.create_eval(...) with custom_llm_provider set to anything other than 'openai' (e.g. 'azure', 'anthropic', 'bedrock'); passing a provider string not in LlmProviders (which raises a ValueError from the enum instead).

Common situations: Teams assuming LiteLLM's unified API covers OpenAI Evals across providers; porting eval workflows from OpenAI to Azure OpenAI and passing custom_llm_provider='azure'; typo'd provider strings.

Related errors

• LIST evals is not supported for {custom_llm_provider}
• GET eval is not supported for {custom_llm_provider}
• UPDATE eval is not supported for {custom_llm_provider}
• DELETE eval is not supported for {custom_llm_provider}
• CANCEL eval is not supported for {custom_llm_provider}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/1fa76c8065854d23. Report an issue: GitHub.