BerriAI/litellm · error · ValueError

DELETE run is not supported for {custom_llm_provider}

Error message

DELETE run is not supported for {custom_llm_provider}

What it means

Thrown by litellm.evals delete_run when the resolved custom_llm_provider has no registered Evals API config (BaseEvalsAPIConfig) in ProviderConfigManager. LiteLLM only implements the Evals API (create run, delete run, etc.) for a small set of providers, currently OpenAI. Passing any other provider string (or letting it default incorrectly) makes get_provider_evals_api_config return None and this ValueError fires before any HTTP request is made.

Source

Thrown at litellm/evals/main.py:1888

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("adelete_run", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"DELETE run is not supported for {custom_llm_provider}")

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

        # Transform request
        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE
        (
            url,
            headers,
            request_body,
        ) = evals_api_provider_config.transform_delete_run_request(
            eval_id=eval_id,
            run_id=run_id,
            api_base=api_base,
            litellm_params=litellm_params,
            headers=headers,
        )

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Set custom_llm_provider='openai' (or use a model string that resolves to OpenAI) — the Evals API is OpenAI-only in LiteLLM
2. Check ProviderConfigManager.get_provider_evals_api_config(provider=litellm.LlmProviders(<your provider>)) returns non-None before calling delete_run
3. If you must delete eval runs for another vendor, call that vendor's evals REST endpoint directly with httpx/requests instead of litellm.evals
4. Upgrade litellm — new providers may gain evals support over time

Example fix

// before
await litellm.evals.adelete_run(run_id=run_id, custom_llm_provider="anthropic")

// after
await litellm.evals.adelete_run(run_id=run_id, custom_llm_provider="openai")

import litellm
from litellm.utils import ProviderConfigManager

cfg = ProviderConfigManager.get_provider_evals_api_config(
    provider=litellm.LlmProviders("anthropic")
)
if cfg is None:
    raise SystemExit("evals delete_run unsupported for this provider; use openai")

try:
    await litellm.evals.adelete_run(run_id=rid, custom_llm_provider=provider)
except ValueError as e:
    if "DELETE run is not supported" in str(e):
        # unsupported provider: skip or call vendor REST API directly
        ...

When it happens

Trigger: Calling litellm.evals.delete_run(...) (or the async variant) with custom_llm_provider set to a provider other than 'openai', e.g. 'anthropic', 'azure', 'bedrock', or with a model string like 'gemini/gemini-2.0-flash' whose provider prefix LiteLLM maps to a non-OpenAI provider.

Common situations: Teams wiring eval-run lifecycle management into CI cleanup steps for a non-OpenAI LLM; copy-pasting evals example code that worked against OpenAI and switching the model; misreading model routing so 'openai/...' vs '<provider>/...' resolves to the wrong provider.

Related errors

• CREATE eval is not supported for {custom_llm_provider}
• LIST evals is not supported for {custom_llm_provider}
• GET eval is not supported for {custom_llm_provider}
• UPDATE eval is not supported for {custom_llm_provider}
• DELETE eval is not supported for {custom_llm_provider}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/774d5a13174b000d. Report an issue: GitHub.