{"record":{"id":"774d5a13174b000d","repo":"BerriAI/litellm","slug":"delete-run-is-not-supported-for-custom-llm-provid","errorCode":null,"errorMessage":"DELETE run is not supported for {custom_llm_provider}","messagePattern":"DELETE run is not supported for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/evals/main.py","lineNumber":1888,"sourceCode":"    try:\n        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get(\"litellm_logging_obj\")\n        litellm_call_id: Final[str | None] = kwargs.get(\"litellm_call_id\", None)\n        _is_async: Final = kwargs.pop(\"adelete_run\", False) is True\n\n        # Get LiteLLM parameters\n        litellm_params: Final = GenericLiteLLMParams(**kwargs)\n\n        # Determine provider\n        if custom_llm_provider is None:\n            custom_llm_provider = \"openai\"\n\n        # Get provider config\n        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(\n            provider=litellm.LlmProviders(custom_llm_provider),\n        )\n\n        if evals_api_provider_config is None:\n            raise ValueError(f\"DELETE run is not supported for {custom_llm_provider}\")\n\n        # Validate environment and get headers\n        headers = extra_headers or {}\n        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)\n\n        # Transform request\n        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE\n        (\n            url,\n            headers,\n            request_body,\n        ) = evals_api_provider_config.transform_delete_run_request(\n            eval_id=eval_id,\n            run_id=run_id,\n            api_base=api_base,\n            litellm_params=litellm_params,\n            headers=headers,\n        )","sourceCodeStart":1870,"sourceCodeEnd":1906,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/evals/main.py#L1870-L1906","documentation":"Thrown by litellm.evals delete_run when the resolved custom_llm_provider has no registered Evals API config (BaseEvalsAPIConfig) in ProviderConfigManager. LiteLLM only implements the Evals API (create run, delete run, etc.) for a small set of providers, currently OpenAI. Passing any other provider string (or letting it default incorrectly) makes get_provider_evals_api_config return None and this ValueError fires before any HTTP request is made.","triggerScenarios":"Calling litellm.evals.delete_run(...) (or the async variant) with custom_llm_provider set to a provider other than 'openai', e.g. 'anthropic', 'azure', 'bedrock', or with a model string like 'gemini/gemini-2.0-flash' whose provider prefix LiteLLM maps to a non-OpenAI provider.","commonSituations":"Teams wiring eval-run lifecycle management into CI cleanup steps for a non-OpenAI LLM; copy-pasting evals example code that worked against OpenAI and switching the model; misreading model routing so 'openai/...' vs '<provider>/...' resolves to the wrong provider.","solutions":["Set custom_llm_provider='openai' (or use a model string that resolves to OpenAI) — the Evals API is OpenAI-only in LiteLLM","Check ProviderConfigManager.get_provider_evals_api_config(provider=litellm.LlmProviders(<your provider>)) returns non-None before calling delete_run","If you must delete eval runs for another vendor, call that vendor's evals REST endpoint directly with httpx/requests instead of litellm.evals","Upgrade litellm — new providers may gain evals support over time"],"exampleFix":"// before\nawait litellm.evals.adelete_run(run_id=run_id, custom_llm_provider=\"anthropic\")\n\n// after\nawait litellm.evals.adelete_run(run_id=run_id, custom_llm_provider=\"openai\")","handlingStrategy":"validation","validationCode":"import litellm\nfrom litellm.utils import ProviderConfigManager\n\ncfg = ProviderConfigManager.get_provider_evals_api_config(\n    provider=litellm.LlmProviders(\"anthropic\")\n)\nif cfg is None:\n    raise SystemExit(\"evals delete_run unsupported for this provider; use openai\")","typeGuard":null,"tryCatchPattern":"try:\n    await litellm.evals.adelete_run(run_id=rid, custom_llm_provider=provider)\nexcept ValueError as e:\n    if \"DELETE run is not supported\" in str(e):\n        # unsupported provider: skip or call vendor REST API directly\n        ...","preventionTips":["Pin custom_llm_provider='openai' for evals calls","Check ProviderConfigManager before calling in generic wrappers"],"tags":["evals","provider-support","validation"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}