BerriAI/litellm · error · ValueError

CREATE run is not supported for {custom_llm_provider}

Error message

CREATE run is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.create_run (litellm/evals/main.py:1213) when the resolved provider has no Evals API config. Run execution (launching an eval against a data source) is OpenAI-only in LiteLLM — get_provider_evals_api_config maps only LlmProviders.OPENAI to OpenAIEvalsConfig — so any other custom_llm_provider is refused before the run request is assembled.

Source

Thrown at litellm/evals/main.py:1213

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("acreate_run", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"CREATE run is not supported for {custom_llm_provider}")

        # Build create request
        create_request: Final[CreateRunRequest] = {
            "data_source": data_source,
        }
        if name is not None:
            create_request["name"] = name
        # if metadata is not None:
        #     create_request["metadata"] = metadata

        # Merge extra_body if provided
        if extra_body:
            create_request.update(extra_body)

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

View on GitHub (pinned to 6c2dcb801b)

Solutions

  1. Run evals with custom_llm_provider='openai' (default) — note the model under test is specified inside testing_criteria/data_source, not via custom_llm_provider
  2. For non-OpenAI evaluation, use LiteLLM completion calls inside your own grading loop or a framework like braintrust/langfuse
  3. Verify OPENAI_API_KEY and api_base are set for the OpenAI Evals endpoint

Example fix

# before
litellm.evals.create_run(eval_id="eval_123", data_source=ds, custom_llm_provider="anthropic")

# after
litellm.evals.create_run(eval_id="eval_123", data_source=ds, custom_llm_provider="openai")
Defensive patterns

Strategy: validation

Validate before calling

if custom_llm_provider != "openai":
    raise ValueError("create_run requires the OpenAI provider")
litellm.evals.create_run(eval_id=eval_id, data_source=ds, custom_llm_provider="openai")

Type guard

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

Try / catch

try:
    litellm.evals.create_run(eval_id=eval_id, data_source=ds)
except ValueError as e:
    if "CREATE run is not supported" in str(e):
        logger.error("eval runs execute on OpenAI; set custom_llm_provider='openai'")
        raise

Prevention

When it happens

Trigger: Calling litellm.evals.create_run(eval_id=..., data_source=..., custom_llm_provider='anthropic'|'azure'|...); executing OpenAI-created evals through a different provider's credentials.

Common situations: Evaluating models on non-OpenAI providers and expecting litellm.evals to run them; passing the provider of the model under test rather than the provider hosting the eval.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/b05883b9821dbdfb. Report an issue: GitHub.