{"record":{"id":"3e7993e48c7896e7","repo":"BerriAI/litellm","slug":"get-run-is-not-supported-for-custom-llm-provider","errorCode":null,"errorMessage":"GET run is not supported for {custom_llm_provider}","messagePattern":"GET run is not supported for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/evals/main.py","lineNumber":1567,"sourceCode":"    try:\n        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get(\"litellm_logging_obj\")\n        litellm_call_id: Final[str | None] = kwargs.get(\"litellm_call_id\", None)\n        _is_async: Final = kwargs.pop(\"aget_run\", False) is True\n\n        # Get LiteLLM parameters\n        litellm_params: Final = GenericLiteLLMParams(**kwargs)\n\n        # Determine provider\n        if custom_llm_provider is None:\n            custom_llm_provider = \"openai\"\n\n        # Get provider config\n        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(\n            provider=litellm.LlmProviders(custom_llm_provider),\n        )\n\n        if evals_api_provider_config is None:\n            raise ValueError(f\"GET run is not supported for {custom_llm_provider}\")\n\n        # Validate environment and get headers\n        headers = extra_headers or {}\n        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)\n\n        # Transform request\n        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE\n        url, headers = evals_api_provider_config.transform_get_run_request(\n            eval_id=eval_id,\n            run_id=run_id,\n            api_base=api_base,\n            litellm_params=litellm_params,\n            headers=headers,\n        )\n\n        # Pre-call logging\n        litellm_logging_obj.update_from_kwargs(\n            kwargs=kwargs,","sourceCodeStart":1549,"sourceCodeEnd":1585,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/evals/main.py#L1549-L1585","documentation":"Raised in litellm.evals.get_run (litellm/evals/main.py:1567) when the provider passed in has no Evals API config. Only LlmProviders.OPENAI is supported by ProviderConfigManager.get_provider_evals_api_config; fetching a specific run by eval_id/run_id against any other provider is rejected before the GET request is transformed.","triggerScenarios":"Calling litellm.evals.get_run(eval_id=..., run_id=..., custom_llm_provider=<non-openai>); polling run status with a provider string copied from the model-under-test configuration.","commonSituations":"Status-polling loops for eval runs; mixed-provider apps where the eval host (OpenAI) differs from the graded model's provider.","solutions":["Use custom_llm_provider='openai' (default) for get_run","Keep eval-host credentials (OPENAI_API_KEY) separate from the model-under-test provider config","Poll non-OpenAI run status via the provider's native API"],"exampleFix":"# before\nrun = litellm.evals.get_run(eval_id=\"eval_123\", run_id=\"run_abc\", custom_llm_provider=\"vertex_ai\")\n\n# after\nrun = litellm.evals.get_run(eval_id=\"eval_123\", run_id=\"run_abc\", custom_llm_provider=\"openai\")","handlingStrategy":"validation","validationCode":"if custom_llm_provider != \"openai\":\n    raise ValueError(\"get_run requires the OpenAI provider\")\nrun = litellm.evals.get_run(eval_id=eval_id, run_id=run_id, custom_llm_provider=\"openai\")","typeGuard":"def evals_provider_supported(provider: str) -> bool:\n    return provider == \"openai\"","tryCatchPattern":"try:\n    run = litellm.evals.get_run(eval_id=eval_id, run_id=run_id)\nexcept ValueError as e:\n    if \"GET run is not supported\" in str(e):\n        logger.error(\"eval run polling is OpenAI-only\")\n        raise","preventionTips":["Separate eval-host credentials (OpenAI) from model-under-test provider config","Pin provider='openai' in polling loops","Surface this as a configuration error, not a transient failure — do not retry"],"tags":["evals","runs","provider-support","openai","validation"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}