BerriAI/litellm · error · ValueError

CANCEL run is not supported for {custom_llm_provider}

Error message

CANCEL run is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.cancel_run (litellm/evals/main.py:1723) when get_provider_evals_api_config finds no config for the provider. Cancelling a specific eval run is OpenAI-only in LiteLLM; any other custom_llm_provider fails before validate_environment or transform_cancel_run_request execute.

Source

Thrown at litellm/evals/main.py:1723

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("acancel_run", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"CANCEL run is not supported for {custom_llm_provider}")

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

        # Transform request
        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE
        (
            url,
            headers,
            request_body,
        ) = evals_api_provider_config.transform_cancel_run_request(
            eval_id=eval_id,
            run_id=run_id,
            api_base=api_base,
            litellm_params=litellm_params,
            headers=headers,
        )

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Call cancel_run with custom_llm_provider='openai' or omit the argument
2. Cancel non-OpenAI runs through the provider's REST API
3. Add a unit assertion that your evals module always uses the 'openai' provider constant

Example fix

# before
litellm.evals.cancel_run(eval_id="eval_123", run_id="run_abc", custom_llm_provider="azure")

# after
litellm.evals.cancel_run(eval_id="eval_123", run_id="run_abc", custom_llm_provider="openai")

if custom_llm_provider != "openai":
    raise ValueError("cancel_run requires the OpenAI provider")
litellm.evals.cancel_run(eval_id=eval_id, run_id=run_id, custom_llm_provider="openai")

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

try:
    litellm.evals.cancel_run(eval_id=eval_id, run_id=run_id)
except ValueError as e:
    if "CANCEL run is not supported" in str(e):
        raise UnsupportedProviderError("run cancellation is OpenAI-only in litellm")

When it happens

Trigger: Calling litellm.evals.cancel_run(eval_id=..., run_id=..., custom_llm_provider=<non-openai>); abort actions wired to generic provider config in an eval orchestration tool.

Common situations: Orchestration platforms (Airflow DAGs, background workers) that pass one provider env var to every litellm call; assuming cancel parity because create_run worked with 'openai' but cancel code reuses a different constant.

Related errors

• CANCEL eval is not supported for {custom_llm_provider}
• CREATE run is not supported for {custom_llm_provider}
• LIST runs is not supported for {custom_llm_provider}
• GET run is not supported for {custom_llm_provider}
• CREATE eval is not supported for {custom_llm_provider}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/234225fea51d1f0d. Report an issue: GitHub.