BerriAI/litellm · error · ValueError

CANCEL eval is not supported for {custom_llm_provider}

Error message

CANCEL eval is not supported for {custom_llm_provider}

What it means

Raised in litellm.evals.cancel_eval (litellm/evals/main.py:1034) when the provider has no Evals API config. ProviderConfigManager.get_provider_evals_api_config returns a config only for LlmProviders.OPENAI; cancelling eval runs through LiteLLM against any other provider is unsupported and rejected upfront.

Source

Thrown at litellm/evals/main.py:1034

    try:
        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get("litellm_logging_obj")
        litellm_call_id: Final[str | None] = kwargs.get("litellm_call_id", None)
        _is_async: Final = kwargs.pop("acancel_eval", False) is True

        # Get LiteLLM parameters
        litellm_params: Final = GenericLiteLLMParams(**kwargs)

        # Determine provider
        if custom_llm_provider is None:
            custom_llm_provider = "openai"

        # Get provider config
        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(
            provider=litellm.LlmProviders(custom_llm_provider),
        )

        if evals_api_provider_config is None:
            raise ValueError(f"CANCEL eval is not supported for {custom_llm_provider}")

        # Validate environment and get headers
        headers = extra_headers or {}
        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)

        # Transform request
        api_base: Final = litellm_params.api_base or DEFAULT_OPENAI_API_BASE
        (
            url,
            headers,
            request_body,
        ) = evals_api_provider_config.transform_cancel_eval_request(
            eval_id=eval_id,
            api_base=api_base,
            litellm_params=litellm_params,
            headers=headers,
        )

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Call cancel_eval with custom_llm_provider='openai' (or omit it)
2. For other providers, cancel runs via their native APIs
3. Wrap eval-ops code in a provider check so non-OpenAI paths skip litellm.evals entirely

Example fix

# before
litellm.evals.cancel_eval(eval_id="eval_123", custom_llm_provider="azure")

# after
litellm.evals.cancel_eval(eval_id="eval_123", custom_llm_provider="openai")

if custom_llm_provider != "openai":
    raise ValueError("cancel_eval requires the OpenAI provider")
litellm.evals.cancel_eval(eval_id=eval_id, custom_llm_provider="openai")

def evals_provider_supported(provider: str) -> bool:
    return provider == "openai"

try:
    litellm.evals.cancel_eval(eval_id=eval_id)
except ValueError as e:
    if "CANCEL eval is not supported" in str(e):
        logger.error("eval cancellation is OpenAI-only in litellm")
        raise

When it happens

Trigger: Calling litellm.evals.cancel_eval(eval_id=..., custom_llm_provider=<non-openai>); long-running eval cancellation workflows pointed at Azure/Anthropic/etc.

Common situations: Ops tooling that cancels runaway evals assuming provider-agnostic support; config templating that injects the deployment provider into all litellm helper calls.

Related errors

• CANCEL run is not supported for {custom_llm_provider}
• CREATE eval is not supported for {custom_llm_provider}
• LIST evals is not supported for {custom_llm_provider}
• GET eval is not supported for {custom_llm_provider}
• UPDATE eval is not supported for {custom_llm_provider}

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/181289b742bdf9d4. Report an issue: GitHub.