{"record":{"id":"93bb585d4f03a725","repo":"headroomlabs-ai/headroom","slug":"unknown-provider-self-llm-config-provider","errorCode":null,"errorMessage":"Unknown provider: {self.llm_config.provider}","messagePattern":"Unknown provider: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/evals/runners/before_after.py","lineNumber":127,"sourceCode":"            try:\n                import openai\n\n                return openai.OpenAI()\n            except ImportError as e:\n                raise ImportError(\n                    \"openai package required. Install with: pip install openai\"\n                ) from e\n        elif self.llm_config.provider == \"ollama\":\n            try:\n                import ollama\n\n                return ollama.Client()\n            except ImportError as e:\n                raise ImportError(\n                    \"ollama package required. Install with: pip install ollama\"\n                ) from e\n        else:\n            raise ValueError(f\"Unknown provider: {self.llm_config.provider}\")\n\n    def _init_proxy_client(self) -> Any:\n        \"\"\"Initialize an OpenAI client pointing at the Headroom proxy.\"\"\"\n        import openai\n\n        return openai.OpenAI(\n            base_url=f\"{self.llm_config.headroom_proxy_url}/v1\",\n            api_key=os.environ.get(\"OPENAI_API_KEY\", \"\"),\n        )\n\n    def _call_llm_via_proxy(self, context: str, query: str) -> str:\n        \"\"\"Call LLM through Headroom proxy (full stack: compression + CCR).\"\"\"\n        prompt = f\"\"\"Based on the following context, answer the question.\n\nContext:\n{context}\n\nQuestion: {query}","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/evals/runners/before_after.py#L109-L145","documentation":"Raised by BeforeAfterRunner._init_llm_client when llm_config.provider does not match any of the supported branches ('anthropic', 'openai', 'ollama'). It is an exhaustive-match guard at the bottom of the provider dispatch: any typo, casing mismatch ('OpenAI'), or future/new provider name that this version of headroom does not know about lands here. The offending value is interpolated into the message.","triggerScenarios":"Setting LLMJudgeConfig/provider in a suite spec or CLI flag to an unsupported string, e.g. 'azure-openai', 'OpenAI', 'azure', 'bedrock', 'google', 'groq', or trailing whitespace 'openai '. Also triggered when a config file written for a newer headroom version (with more providers) is run under an older install.","commonSituations":"Hand-edited eval YAML with a provider typo; version drift between the headroom that documented a provider and the one installed; copy-pasting provider names from other tools (LiteLLM-style names like 'gpt-4o' or 'claude-3' passed as provider).","solutions":["Set provider to one of the exact supported strings: 'anthropic', 'openai', or 'ollama' (lowercase).","Print/validate the config before running (e.g. dump the parsed LLMJudgeConfig) to catch typos and whitespace.","If you need another backend, check for a newer headroom release that added it, or extend _init_llm_client locally.","Note model names are separate from provider — put 'gpt-4o' in the model field, not provider."],"exampleFix":"# before\nllm_config = LLMJudgeConfig(provider=\"OpenAI\")  # -> Unknown provider: OpenAI\n\n# after\nllm_config = LLMJudgeConfig(provider=\"openai\", model=\"gpt-4o\")","handlingStrategy":"validation","validationCode":"SUPPORTED_PROVIDERS = {\"anthropic\", \"openai\", \"ollama\"}\n\nprovider = llm_config.provider.strip().lower()\nif provider not in SUPPORTED_PROVIDERS:\n    raise SystemExit(\n        f\"provider must be one of {sorted(SUPPORTED_PROVIDERS)}, got {llm_config.provider!r}\"\n    )","typeGuard":"def is_supported_provider(p: str) -> bool:\n    \"\"\"Narrow provider strings to the ones BeforeAfterRunner dispatches on.\"\"\"\n    return isinstance(p, str) and p in {\"anthropic\", \"openai\", \"ollama\"}","tryCatchPattern":"try:\n    runner.run()\nexcept ValueError as e:\n    if str(e).startswith(\"Unknown provider\"):\n        sys.exit(f\"fix provider in suite config: {e}\")\n    raise","preventionTips":["Validate provider against the supported set where you load the config, not deep in the run.","Keep provider names lowercase in all suite files; treat the list as an enum.","On headroom upgrades, re-check the supported-provider list — new providers may have been added."],"tags":["configuration","validation","provider","evals","typo"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}