What causes "DELETE eval is not supported for {custom_llm_provider}" in BerriAI/litellm?

Use custom_llm_provider='openai' or omit the argument for delete_eval

BerriAI/litellm · error · ValueError

DELETE eval is not supported for {custom_llm_provider}

Error message

DELETE eval is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.delete_eval (litellm/evals/main.py:884) when get_provider_evals_api_config returns None for the given provider. Only LlmProviders.OPENAI has an Evals config, so DELETE operations via litellm.evals are OpenAI-only; anything else fails before headers/URL transformation.

Source

Thrown at litellm/evals/main.py:884

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("adelete_eval", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"DELETE eval is not supported for {custom_llm_provider}")

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

        # Transform request
        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE
        url, headers = evals_api_provider_config.transform_delete_eval_request(
            eval_id=eval_id,
            api_base=api_base,
            litellm_params=litellm_params,
            headers=headers,
        )

        # Pre-call logging
        litellm_logging_obj.update_from_kwargs(
            kwargs=kwargs,
            model=None,

View on GitHub (pinned to 6c2dcb801b)

Solutions

Use custom_llm_provider='openai' or omit the argument for delete_eval
Delete evals through the provider's own API when not using OpenAI
Add a preflight check in cleanup tooling that skips non-OpenAI providers for eval deletion

Example fix

# before
litellm.evals.delete_eval(eval_id="eval_123", custom_llm_provider="bedrock")

# after
litellm.evals.delete_eval(eval_id="eval_123", custom_llm_provider="openai")

Defensive patterns

Strategy: validation

Validate before calling

if custom_llm_provider != "openai":
    logger.warning("skipping eval deletion for unsupported provider %s", custom_llm_provider)
else:
    litellm.evals.delete_eval(eval_id=eval_id, custom_llm_provider="openai")

Type guard

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

Try / catch

try:
    litellm.evals.delete_eval(eval_id=eval_id)
except ValueError as e:
    if "DELETE eval is not supported" in str(e):
        raise UnsupportedProviderError("eval deletion is OpenAI-only")

Prevention

Skip delete ops for non-OpenAI providers in teardown scripts instead of calling through
Use a single provider allowlist check across all eval lifecycle calls
Keep deletion idempotent so partial cleanup can be re-run

When it happens

Trigger: Calling litellm.evals.delete_eval(eval_id=..., custom_llm_provider='azure'|'anthropic'|'vertex_ai'|...); cleanup scripts parameterized by provider that pass a non-OpenAI value.

Common situations: Automated teardown scripts iterating over providers; porting eval lifecycle management from the OpenAI SDK to LiteLLM with a different provider configured.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/7bd61d5bfe987d24. Report an issue: GitHub.