{"record":{"id":"b05883b9821dbdfb","repo":"BerriAI/litellm","slug":"create-run-is-not-supported-for-custom-llm-provid","errorCode":null,"errorMessage":"CREATE run is not supported for {custom_llm_provider}","messagePattern":"CREATE run is not supported for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/evals/main.py","lineNumber":1213,"sourceCode":"    try:\n        litellm_logging_obj: Final[LiteLLMLoggingObj] = kwargs.get(\"litellm_logging_obj\")\n        litellm_call_id: Final[str | None] = kwargs.get(\"litellm_call_id\", None)\n        _is_async: Final = kwargs.pop(\"acreate_run\", False) is True\n\n        # Get LiteLLM parameters\n        litellm_params: Final = GenericLiteLLMParams(**kwargs)\n\n        # Determine provider\n        if custom_llm_provider is None:\n            custom_llm_provider = \"openai\"\n\n        # Get provider config\n        evals_api_provider_config: BaseEvalsAPIConfig | None = ProviderConfigManager.get_provider_evals_api_config(\n            provider=litellm.LlmProviders(custom_llm_provider),\n        )\n\n        if evals_api_provider_config is None:\n            raise ValueError(f\"CREATE run is not supported for {custom_llm_provider}\")\n\n        # Build create request\n        create_request: Final[CreateRunRequest] = {\n            \"data_source\": data_source,\n        }\n        if name is not None:\n            create_request[\"name\"] = name\n        # if metadata is not None:\n        #     create_request[\"metadata\"] = metadata\n\n        # Merge extra_body if provided\n        if extra_body:\n            create_request.update(extra_body)\n\n        # Validate environment and get headers\n        headers = extra_headers or {}\n        headers = evals_api_provider_config.validate_environment(headers=headers, litellm_params=litellm_params)\n","sourceCodeStart":1195,"sourceCodeEnd":1231,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/evals/main.py#L1195-L1231","documentation":"Raised in litellm.evals.create_run (litellm/evals/main.py:1213) when the resolved provider has no Evals API config. Run execution (launching an eval against a data source) is OpenAI-only in LiteLLM — get_provider_evals_api_config maps only LlmProviders.OPENAI to OpenAIEvalsConfig — so any other custom_llm_provider is refused before the run request is assembled.","triggerScenarios":"Calling litellm.evals.create_run(eval_id=..., data_source=..., custom_llm_provider='anthropic'|'azure'|...); executing OpenAI-created evals through a different provider's credentials.","commonSituations":"Evaluating models on non-OpenAI providers and expecting litellm.evals to run them; passing the provider of the model under test rather than the provider hosting the eval.","solutions":["Run evals with custom_llm_provider='openai' (default) — note the model under test is specified inside testing_criteria/data_source, not via custom_llm_provider","For non-OpenAI evaluation, use LiteLLM completion calls inside your own grading loop or a framework like braintrust/langfuse","Verify OPENAI_API_KEY and api_base are set for the OpenAI Evals endpoint"],"exampleFix":"# before\nlitellm.evals.create_run(eval_id=\"eval_123\", data_source=ds, custom_llm_provider=\"anthropic\")\n\n# after\nlitellm.evals.create_run(eval_id=\"eval_123\", data_source=ds, custom_llm_provider=\"openai\")","handlingStrategy":"validation","validationCode":"if custom_llm_provider != \"openai\":\n    raise ValueError(\"create_run requires the OpenAI provider\")\nlitellm.evals.create_run(eval_id=eval_id, data_source=ds, custom_llm_provider=\"openai\")","typeGuard":"def evals_provider_supported(provider: str) -> bool:\n    return provider == \"openai\"","tryCatchPattern":"try:\n    litellm.evals.create_run(eval_id=eval_id, data_source=ds)\nexcept ValueError as e:\n    if \"CREATE run is not supported\" in str(e):\n        logger.error(\"eval runs execute on OpenAI; set custom_llm_provider='openai'\")\n        raise","preventionTips":["Remember custom_llm_provider selects the eval HOST, not the model under test","Use a dedicated eval harness (own grading loop) for non-OpenAI model evaluation","Assert provider == 'openai' in a preflight check inside eval orchestration code"],"tags":["evals","runs","provider-support","openai","validation"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}