{"record":{"id":"0bc6cc0a3d0ea4ef","repo":"BerriAI/litellm","slug":"list-evals-is-not-supported-for-custom-llm-provid","errorCode":null,"errorMessage":"LIST evals is not supported for {custom_llm_provider}","messagePattern":"LIST evals is not supported for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/evals/main.py","lineNumber":348,"sourceCode":"    try:\n        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get(\"litellm_logging_obj\")\n        litellm_call_id: Final[str | None] = kwargs.get(\"litellm_call_id\", None)\n        _is_async: Final = kwargs.pop(\"alist_evals\", False) is True\n\n        # Get LiteLLM parameters\n        litellm_params: Final = GenericLiteLLMParams(**kwargs)\n\n        # Determine provider\n        if custom_llm_provider is None:\n            custom_llm_provider = \"openai\"\n\n        # Get provider config\n        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(\n            provider=litellm.LlmProviders(custom_llm_provider),\n        )\n\n        if evals_api_provider_config is None:\n            raise ValueError(f\"LIST evals is not supported for {custom_llm_provider}\")\n\n        # Build list parameters\n        list_params: Final[ListEvalsParams] = {}\n        if limit is not None:\n            list_params[\"limit\"] = limit\n        if after is not None:\n            list_params[\"after\"] = after\n        if before is not None:\n            list_params[\"before\"] = before\n        if order is not None:\n            list_params[\"order\"] = order\n        if order_by is not None:\n            list_params[\"order_by\"] = order_by\n\n        # Merge extra_query if provided\n        if extra_query:\n            list_params.update(extra_query)\n","sourceCodeStart":330,"sourceCodeEnd":366,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/evals/main.py#L330-L366","documentation":"Raised in litellm.evals.list_evals (litellm/evals/main.py:348) when the resolved provider has no Evals API config. ProviderConfigManager.get_provider_evals_api_config only returns a config for LlmProviders.OPENAI; every other provider yields None and triggers this error before any list parameters are applied.","triggerScenarios":"Calling litellm.evals.list_evals(custom_llm_provider='anthropic'|'azure'|'bedrock'|...) or any non-'openai' value; omitting the provider is safe because it defaults to 'openai'.","commonSituations":"Switching an eval-dashboard integration from OpenAI to another provider while keeping litellm.evals calls; copy-pasted provider strings from completion() code into evals calls.","solutions":["Use custom_llm_provider='openai' (or omit the argument) so the OpenAI Evals config is selected","Query the target provider's eval API directly via HTTP if you need non-OpenAI eval listings","Check litellm/utils.py get_provider_evals_api_config for the currently supported provider list before upgrading assumptions"],"exampleFix":"# before\nevals = litellm.evals.list_evals(custom_llm_provider=\"azure\")\n\n# after\nevals = litellm.evals.list_evals(custom_llm_provider=\"openai\")","handlingStrategy":"validation","validationCode":"if custom_llm_provider != \"openai\":\n    raise ValueError(\"listing evals requires custom_llm_provider='openai'\")\nlitellm.evals.list_evals(custom_llm_provider=\"openai\", ...)","typeGuard":"def evals_provider_supported(provider: str) -> bool:\n    return provider == \"openai\"","tryCatchPattern":"try:\n    litellm.evals.list_evals(...)\nexcept ValueError as e:\n    if \"LIST evals is not supported\" in str(e):\n        return []  # or surface 'unsupported provider' to the UI","preventionTips":["Omit custom_llm_provider in evals calls so the 'openai' default applies","Wrap eval-dashboard queries with a provider allowlist check","Add a unit test asserting your evals module never passes a non-OpenAI provider"],"tags":["evals","provider-support","openai","validation"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}