headroomlabs-ai/headroom · error · ValueError

Unknown provider: {self.provider}

Error message

Unknown provider: {self.provider}

What it means

The terminal else of _init_llm_client(): the eval driver was constructed with a provider string other than 'anthropic' or 'openai'. Provider selection is a free-form string on the driver (only mapped to a default model via a dict .get with an anthropic fallback), so invalid names pass through construction and explode here at client initialization.

Source

Thrown at headroom/evals/batch_compression_eval.py:1051

            try:
                import anthropic

                return anthropic.Anthropic()
            except ImportError as e:
                raise ImportError(
                    "anthropic package required. Install with: pip install anthropic"
                ) from e
        elif self.provider == "openai":
            try:
                import openai

                return openai.OpenAI()
            except ImportError as e:
                raise ImportError(
                    "openai package required. Install with: pip install openai"
                ) from e
        else:
            raise ValueError(f"Unknown provider: {self.provider}")

    def _call_llm(self, messages: list[dict[str, Any]]) -> str:
        """Call LLM and return response text."""
        if self.provider == "anthropic":
            response = self._llm_client.messages.create(
                model=self.model,
                max_tokens=1024,
                temperature=0.0,
                messages=messages,
            )
            return str(response.content[0].text)
        elif self.provider == "openai":
            response = self._llm_client.chat.completions.create(
                model=self.model,
                max_tokens=1024,
                temperature=0.0,
                messages=messages,
            )

View on GitHub (pinned to 322425c43b)

Solutions

1. Use exactly 'anthropic' or 'openai' — lowercase — as the provider value
2. Validate provider at config load: reject values outside {'anthropic','openai'} before constructing the driver, so the failure happens at the boundary, not deep in init
3. If you need another backend, point provider='openai' at an OpenAI-compatible base URL rather than inventing a provider name

Example fix

# before
runner = BatchCompressionEval(provider="Azure", ...)  # ValueError later

# after
PROVIDERS = {"anthropic", "openai"}
provider = provider.lower().strip()
if provider not in PROVIDERS:
    raise ValueError(f"provider must be one of {sorted(PROVIDERS)}, got {provider!r}")
runner = BatchCompressionEval(provider=provider, ...)

ALLOWED_PROVIDERS = {"anthropic", "openai"}

provider = provider.strip().lower()
if provider not in ALLOWED_PROVIDERS:
    raise ValueError(
        f"provider must be one of {sorted(ALLOWED_PROVIDERS)}, got {provider!r}"
    )

def is_supported_provider(value: object) -> bool:
    return isinstance(value, str) and value in {"anthropic", "openai"}

When it happens

Trigger: Passing provider='Anthropic' (capitalized), 'azure', 'bedrock', 'ollama', 'openrouter', or any typo to the batch compression eval driver — the default-model dict silently falls back to claude-sonnet-4-20250514 (via .get), masking the bad value until _init_llm_client raises.

Common situations: Config-driven eval runs reading provider from YAML with casing/typo issues; users assuming extra providers (azure/openrouter) are supported because no validation happened at constructor time; stale configs after a provider rename.

Related errors

• Unknown provider: {self.llm_config.provider}
• position must be one of {POSITIONS}, got {position!r}
• anthropic package required. Install with: pip install anthro
• openai package required. Install with: pip install openai
• OPENAI_API_KEY environment variable required

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/9303f1da1968aec0. Report an issue: GitHub.