headroomlabs-ai/headroom · error · ValueError

Unknown provider: {self.llm_config.provider}

Error message

Unknown provider: {self.llm_config.provider}

What it means

Raised by BeforeAfterRunner._init_llm_client when llm_config.provider does not match any of the supported branches ('anthropic', 'openai', 'ollama'). It is an exhaustive-match guard at the bottom of the provider dispatch: any typo, casing mismatch ('OpenAI'), or future/new provider name that this version of headroom does not know about lands here. The offending value is interpolated into the message.

Source

Thrown at headroom/evals/runners/before_after.py:127

            try:
                import openai

                return openai.OpenAI()
            except ImportError as e:
                raise ImportError(
                    "openai package required. Install with: pip install openai"
                ) from e
        elif self.llm_config.provider == "ollama":
            try:
                import ollama

                return ollama.Client()
            except ImportError as e:
                raise ImportError(
                    "ollama package required. Install with: pip install ollama"
                ) from e
        else:
            raise ValueError(f"Unknown provider: {self.llm_config.provider}")

    def _init_proxy_client(self) -> Any:
        """Initialize an OpenAI client pointing at the Headroom proxy."""
        import openai

        return openai.OpenAI(
            base_url=f"{self.llm_config.headroom_proxy_url}/v1",
            api_key=os.environ.get("OPENAI_API_KEY", ""),
        )

    def _call_llm_via_proxy(self, context: str, query: str) -> str:
        """Call LLM through Headroom proxy (full stack: compression + CCR)."""
        prompt = f"""Based on the following context, answer the question.

Context:
{context}

Question: {query}

View on GitHub (pinned to 322425c43b)

Solutions

1. Set provider to one of the exact supported strings: 'anthropic', 'openai', or 'ollama' (lowercase).
2. Print/validate the config before running (e.g. dump the parsed LLMJudgeConfig) to catch typos and whitespace.
3. If you need another backend, check for a newer headroom release that added it, or extend _init_llm_client locally.
4. Note model names are separate from provider — put 'gpt-4o' in the model field, not provider.

Example fix

# before
llm_config = LLMJudgeConfig(provider="OpenAI")  # -> Unknown provider: OpenAI

# after
llm_config = LLMJudgeConfig(provider="openai", model="gpt-4o")

SUPPORTED_PROVIDERS = {"anthropic", "openai", "ollama"}

provider = llm_config.provider.strip().lower()
if provider not in SUPPORTED_PROVIDERS:
    raise SystemExit(
        f"provider must be one of {sorted(SUPPORTED_PROVIDERS)}, got {llm_config.provider!r}"
    )

def is_supported_provider(p: str) -> bool:
    """Narrow provider strings to the ones BeforeAfterRunner dispatches on."""
    return isinstance(p, str) and p in {"anthropic", "openai", "ollama"}

try:
    runner.run()
except ValueError as e:
    if str(e).startswith("Unknown provider"):
        sys.exit(f"fix provider in suite config: {e}")
    raise

When it happens

Trigger: Setting LLMJudgeConfig/provider in a suite spec or CLI flag to an unsupported string, e.g. 'azure-openai', 'OpenAI', 'azure', 'bedrock', 'google', 'groq', or trailing whitespace 'openai '. Also triggered when a config file written for a newer headroom version (with more providers) is run under an older install.

Common situations: Hand-edited eval YAML with a provider typo; version drift between the headroom that documented a provider and the one installed; copy-pasting provider names from other tools (LiteLLM-style names like 'gpt-4o' or 'claude-3' passed as provider).

Related errors

• Unknown provider: {self.provider}
• Unknown compression-only benchmark: {spec.name}
• bedrock_eventstream_parse_failed
• position must be one of {POSITIONS}, got {position!r}
• OpenAI API key required. Set OPENAI_API_KEY environment vari

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/93bb585d4f03a725. Report an issue: GitHub.