BerriAI/litellm · error · ValueError

GET run is not supported for {custom_llm_provider}

Error message

GET run is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.get_run (litellm/evals/main.py:1567) when the provider passed in has no Evals API config. Only LlmProviders.OPENAI is supported by ProviderConfigManager.get_provider_evals_api_config; fetching a specific run by eval_id/run_id against any other provider is rejected before the GET request is transformed.

Source

Thrown at litellm/evals/main.py:1567

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("aget_run", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"GET run is not supported for {custom_llm_provider}")

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

        # Transform request
        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE
        url, headers = evals_api_provider_config.transform_get_run_request(
            eval_id=eval_id,
            run_id=run_id,
            api_base=api_base,
            litellm_params=litellm_params,
            headers=headers,
        )

        # Pre-call logging
        litellm_logging_obj.update_from_kwargs(
            kwargs=kwargs,

View on GitHub (pinned to 6c2dcb801b)

Solutions

  1. Use custom_llm_provider='openai' (default) for get_run
  2. Keep eval-host credentials (OPENAI_API_KEY) separate from the model-under-test provider config
  3. Poll non-OpenAI run status via the provider's native API

Example fix

# before
run = litellm.evals.get_run(eval_id="eval_123", run_id="run_abc", custom_llm_provider="vertex_ai")

# after
run = litellm.evals.get_run(eval_id="eval_123", run_id="run_abc", custom_llm_provider="openai")
Defensive patterns

Strategy: validation

Validate before calling

if custom_llm_provider != "openai":
    raise ValueError("get_run requires the OpenAI provider")
run = litellm.evals.get_run(eval_id=eval_id, run_id=run_id, custom_llm_provider="openai")

Type guard

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

Try / catch

try:
    run = litellm.evals.get_run(eval_id=eval_id, run_id=run_id)
except ValueError as e:
    if "GET run is not supported" in str(e):
        logger.error("eval run polling is OpenAI-only")
        raise

Prevention

When it happens

Trigger: Calling litellm.evals.get_run(eval_id=..., run_id=..., custom_llm_provider=<non-openai>); polling run status with a provider string copied from the model-under-test configuration.

Common situations: Status-polling loops for eval runs; mixed-provider apps where the eval host (OpenAI) differs from the graded model's provider.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/3e7993e48c7896e7. Report an issue: GitHub.