{"record":{"id":"f9e624865b013039","repo":"BerriAI/litellm","slug":"list-runs-is-not-supported-for-custom-llm-provide","errorCode":null,"errorMessage":"LIST runs is not supported for {custom_llm_provider}","messagePattern":"LIST runs is not supported for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/evals/main.py","lineNumber":1397,"sourceCode":"    try:\n        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get(\"litellm_logging_obj\")\n        litellm_call_id: Final[str | None] = kwargs.get(\"litellm_call_id\", None)\n        _is_async: Final = kwargs.pop(\"alist_runs\", False) is True\n\n        # Get LiteLLM parameters\n        litellm_params: Final = GenericLiteLLMParams(**kwargs)\n\n        # Determine provider\n        if custom_llm_provider is None:\n            custom_llm_provider = \"openai\"\n\n        # Get provider config\n        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(\n            provider=litellm.LlmProviders(custom_llm_provider),\n        )\n\n        if evals_api_provider_config is None:\n            raise ValueError(f\"LIST runs is not supported for {custom_llm_provider}\")\n\n        # Build list parameters\n        list_params: Final[ListRunsParams] = {}\n        if limit is not None:\n            list_params[\"limit\"] = limit\n        if after is not None:\n            list_params[\"after\"] = after\n        if before is not None:\n            list_params[\"before\"] = before\n        if order is not None:\n            list_params[\"order\"] = order\n\n        # Merge extra_query if provided\n        if extra_query:\n            list_params.update(extra_query)\n\n        # Validate environment and get headers\n        headers = extra_headers or {}","sourceCodeStart":1379,"sourceCodeEnd":1415,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/evals/main.py#L1379-L1415","documentation":"Raised in litellm.evals.list_runs (litellm/evals/main.py:1397) when get_provider_evals_api_config returns None for the provider. Listing runs of an eval is implemented only for OpenAI; every other provider string fails here before pagination parameters (limit/after/before/order) are applied.","triggerScenarios":"Calling litellm.evals.list_runs(eval_id=..., custom_llm_provider=<non-openai>); monitoring dashboards parameterized by deployment provider.","commonSituations":"Building eval observability UIs with LiteLLM and assuming cross-provider run listings; leftover provider values in shared config objects.","solutions":["Call list_runs with custom_llm_provider='openai' or omit the argument","Query run history from the provider's own API for non-OpenAI setups","Centralize the evals provider as a constant ('openai') in your codebase to avoid drift"],"exampleFix":"# before\nruns = litellm.evals.list_runs(eval_id=\"eval_123\", custom_llm_provider=\"azure\")\n\n# after\nruns = litellm.evals.list_runs(eval_id=\"eval_123\", custom_llm_provider=\"openai\")","handlingStrategy":"validation","validationCode":"if custom_llm_provider != \"openai\":\n    raise ValueError(\"list_runs requires the OpenAI provider\")\nruns = litellm.evals.list_runs(eval_id=eval_id, custom_llm_provider=\"openai\")","typeGuard":"def evals_provider_supported(provider: str) -> bool:\n    return provider == \"openai\"","tryCatchPattern":"try:\n    runs = litellm.evals.list_runs(eval_id=eval_id)\nexcept ValueError as e:\n    if \"LIST runs is not supported\" in str(e):\n        return []  # unsupported provider: no run data available","preventionTips":["Omit custom_llm_provider so the OpenAI default applies","In dashboards, gate run-listing views on the provider allowlist","Cache the supported-provider check once per process instead of per request"],"tags":["evals","runs","provider-support","openai","validation"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}