{"record":{"id":"1fa76c8065854d23","repo":"BerriAI/litellm","slug":"create-eval-is-not-supported-for-custom-llm-provi","errorCode":null,"errorMessage":"CREATE eval is not supported for {custom_llm_provider}","messagePattern":"CREATE eval is not supported for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/evals/main.py","lineNumber":163,"sourceCode":"    try:\n        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get(\"litellm_logging_obj\")\n        litellm_call_id: Final[str | None] = kwargs.get(\"litellm_call_id\", None)\n        _is_async: Final = kwargs.pop(\"acreate_eval\", False) is True\n\n        # Get LiteLLM parameters\n        litellm_params: Final = GenericLiteLLMParams(**kwargs)\n\n        # Determine provider\n        if custom_llm_provider is None:\n            custom_llm_provider = \"openai\"\n\n        # Get provider config\n        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(\n            provider=litellm.LlmProviders(custom_llm_provider),\n        )\n\n        if evals_api_provider_config is None:\n            raise ValueError(f\"CREATE eval is not supported for {custom_llm_provider}\")\n\n        # Build create request\n        create_request: Final[CreateEvalRequest] = {\n            \"data_source_config\": data_source_config,\n            \"testing_criteria\": testing_criteria,\n        }\n        if name is not None:\n            create_request[\"name\"] = name\n\n        # Merge extra_body if provided\n        if extra_body:\n            create_request.update(extra_body)\n\n        # Validate environment and get headers\n        headers = extra_headers or {}\n        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)\n\n        # Transform request","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/evals/main.py#L145-L181","documentation":"Raised in litellm.evals.create_eval (litellm/evals/main.py:163) when ProviderConfigManager.get_provider_evals_api_config returns None for the resolved provider. LiteLLM's Evals API bindings currently support only OpenAI (see get_provider_evals_api_config in litellm/utils.py, which returns OpenAIEvalsConfig for LlmProviders.OPENAI and None otherwise), so any other custom_llm_provider is rejected before a request is built.","triggerScenarios":"Calling litellm.evals.create_eval(...) with custom_llm_provider set to anything other than 'openai' (e.g. 'azure', 'anthropic', 'bedrock'); passing a provider string not in LlmProviders (which raises a ValueError from the enum instead).","commonSituations":"Teams assuming LiteLLM's unified API covers OpenAI Evals across providers; porting eval workflows from OpenAI to Azure OpenAI and passing custom_llm_provider='azure'; typo'd provider strings.","solutions":["Pass custom_llm_provider='openai' (or omit it — it defaults to 'openai') and use an OpenAI API key","For Azure OpenAI, point api_base at your Azure endpoint with the openai provider if your setup requires it, or call the Azure Evals REST endpoint directly","Track LiteLLM release notes for new evals provider configs before assuming support"],"exampleFix":"# before\nlitellm.evals.create_eval(custom_llm_provider=\"azure\", ...)\n\n# after\nlitellm.evals.create_eval(custom_llm_provider=\"openai\", ...)","handlingStrategy":"validation","validationCode":"import litellm\nSUPPORTED_EVALS_PROVIDERS = {litellm.LlmProviders.OPENAI.value}\nif custom_llm_provider not in SUPPORTED_EVALS_PROVIDERS:\n    raise ValueError(\"litellm.evals supports only 'openai'\")","typeGuard":"def evals_provider_supported(provider: str) -> bool:\n    return provider == \"openai\"","tryCatchPattern":"try:\n    litellm.evals.create_eval(...)\nexcept ValueError as e:\n    if \"CREATE eval is not supported\" in str(e):\n        logger.error(\"Use custom_llm_provider='openai' for evals\")\n        raise","preventionTips":["Hardcode custom_llm_provider='openai' (or omit it) for all litellm.evals calls","Keep evals provider config separate from model-under-test provider config","Check get_provider_evals_api_config in litellm/utils.py for supported providers at your version"],"tags":["evals","provider-support","openai","validation"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}