{"record":{"id":"9303f1da1968aec0","repo":"headroomlabs-ai/headroom","slug":"unknown-provider-self-provider","errorCode":null,"errorMessage":"Unknown provider: {self.provider}","messagePattern":"Unknown provider: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/evals/batch_compression_eval.py","lineNumber":1051,"sourceCode":"            try:\n                import anthropic\n\n                return anthropic.Anthropic()\n            except ImportError as e:\n                raise ImportError(\n                    \"anthropic package required. Install with: pip install anthropic\"\n                ) from e\n        elif self.provider == \"openai\":\n            try:\n                import openai\n\n                return openai.OpenAI()\n            except ImportError as e:\n                raise ImportError(\n                    \"openai package required. Install with: pip install openai\"\n                ) from e\n        else:\n            raise ValueError(f\"Unknown provider: {self.provider}\")\n\n    def _call_llm(self, messages: list[dict[str, Any]]) -> str:\n        \"\"\"Call LLM and return response text.\"\"\"\n        if self.provider == \"anthropic\":\n            response = self._llm_client.messages.create(\n                model=self.model,\n                max_tokens=1024,\n                temperature=0.0,\n                messages=messages,\n            )\n            return str(response.content[0].text)\n        elif self.provider == \"openai\":\n            response = self._llm_client.chat.completions.create(\n                model=self.model,\n                max_tokens=1024,\n                temperature=0.0,\n                messages=messages,\n            )","sourceCodeStart":1033,"sourceCodeEnd":1069,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/evals/batch_compression_eval.py#L1033-L1069","documentation":"The terminal else of _init_llm_client(): the eval driver was constructed with a provider string other than 'anthropic' or 'openai'. Provider selection is a free-form string on the driver (only mapped to a default model via a dict .get with an anthropic fallback), so invalid names pass through construction and explode here at client initialization.","triggerScenarios":"Passing provider='Anthropic' (capitalized), 'azure', 'bedrock', 'ollama', 'openrouter', or any typo to the batch compression eval driver — the default-model dict silently falls back to claude-sonnet-4-20250514 (via .get), masking the bad value until _init_llm_client raises.","commonSituations":"Config-driven eval runs reading provider from YAML with casing/typo issues; users assuming extra providers (azure/openrouter) are supported because no validation happened at constructor time; stale configs after a provider rename.","solutions":["Use exactly 'anthropic' or 'openai' — lowercase — as the provider value","Validate provider at config load: reject values outside {'anthropic','openai'} before constructing the driver, so the failure happens at the boundary, not deep in init","If you need another backend, point provider='openai' at an OpenAI-compatible base URL rather than inventing a provider name"],"exampleFix":"# before\nrunner = BatchCompressionEval(provider=\"Azure\", ...)  # ValueError later\n\n# after\nPROVIDERS = {\"anthropic\", \"openai\"}\nprovider = provider.lower().strip()\nif provider not in PROVIDERS:\n    raise ValueError(f\"provider must be one of {sorted(PROVIDERS)}, got {provider!r}\")\nrunner = BatchCompressionEval(provider=provider, ...)","handlingStrategy":"validation","validationCode":"ALLOWED_PROVIDERS = {\"anthropic\", \"openai\"}\n\nprovider = provider.strip().lower()\nif provider not in ALLOWED_PROVIDERS:\n    raise ValueError(\n        f\"provider must be one of {sorted(ALLOWED_PROVIDERS)}, got {provider!r}\"\n    )","typeGuard":"def is_supported_provider(value: object) -> bool:\n    return isinstance(value, str) and value in {\"anthropic\", \"openai\"}","tryCatchPattern":null,"preventionTips":["Validate provider at the config boundary, not at client init — the driver's default-model fallback masks typos until late","Normalize casing/whitespace on provider strings from YAML/CLI input","For OpenAI-compatible backends, keep provider='openai' and override the base URL instead of new names"],"tags":["validation","evals","llm","provider","config"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}