headroomlabs-ai/headroom · error · ValueError

Unknown provider

Error message

Unknown provider: {self.llm_config.provider}

What it means

Raised by BeforeAfterRunner._init_llm_client when llm_config.provider does not match any of the supported branches ('anthropic', 'openai', 'ollama'). It is an exhaustive-match guard at the bottom of the provider dispatch: any typo, casing mismatch ('OpenAI'), or future/new provider name that this version of headroom does not know about lands here. The offending value is interpolated into the message.

Solutions

1. Set provider to one of the exact supported strings: 'anthropic', 'openai', or 'ollama' (lowercase).
2. Print/validate the config before running (e.g. dump the parsed LLMJudgeConfig) to catch typos and whitespace.
3. If you need another backend, check for a newer headroom release that added it, or extend _init_llm_client locally.
4. Note model names are separate from provider — put 'gpt-4o' in the model field, not provider.

Example fix

# before
llm_config = LLMJudgeConfig(provider="OpenAI")  # -> Unknown provider: OpenAI

# after
llm_config = LLMJudgeConfig(provider="openai", model="gpt-4o")

SUPPORTED_PROVIDERS = {"anthropic", "openai", "ollama"}

provider = llm_config.provider.strip().lower()
if provider not in SUPPORTED_PROVIDERS:
    raise SystemExit(
        f"provider must be one of {sorted(SUPPORTED_PROVIDERS)}, got {llm_config.provider!r}"
    )

def is_supported_provider(p: str) -> bool:
    """Narrow provider strings to the ones BeforeAfterRunner dispatches on."""
    return isinstance(p, str) and p in {"anthropic", "openai", "ollama"}

try:
    runner.run()
except ValueError as e:
    if str(e).startswith("Unknown provider"):
        sys.exit(f"fix provider in suite config: {e}")
    raise

When it happens

Trigger: Setting LLMJudgeConfig/provider in a suite spec or CLI flag to an unsupported string, e.g. 'azure-openai', 'OpenAI', 'azure', 'bedrock', 'google', 'groq', or trailing whitespace 'openai '. Also triggered when a config file written for a newer headroom version (with more providers) is run under an older install.

Common situations: Hand-edited eval YAML with a provider typo; version drift between the headroom that documented a provider and the one installed; copy-pasting provider names from other tools (LiteLLM-style names like 'gpt-4o' or 'claude-3' passed as provider).

Related errors

• Unknown compression-only benchmark
• Unknown provider
• anthropic package required. Install with: pip install…
• auth_style must be 'post' or 'basic'
• bedrock_eventstream_parse_failed

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/93bb585d4f03a725. Report an issue: GitHub.

Appendix: source

Thrown at headroom/evals/runners/before_after.py:127

            try:
                import openai

                return openai.OpenAI()
            except ImportError as e:
                raise ImportError(
                    "openai package required. Install with: pip install openai"
                ) from e
        elif self.llm_config.provider == "ollama":
            try:
                import ollama

                return ollama.Client()
            except ImportError as e:
                raise ImportError(
                    "ollama package required. Install with: pip install ollama"
                ) from e
        else:
            raise ValueError(f"Unknown provider: {self.llm_config.provider}")

    def _init_proxy_client(self) -> Any:
        """Initialize an OpenAI client pointing at the Headroom proxy."""
        import openai

        return openai.OpenAI(
            base_url=f"{self.llm_config.headroom_proxy_url}/v1",
            api_key=os.environ.get("OPENAI_API_KEY", ""),
        )

    def _call_llm_via_proxy(self, context: str, query: str) -> str:
        """Call LLM through Headroom proxy (full stack: compression + CCR)."""
        prompt = f"""Based on the following context, answer the question.

Context:
{context}

Question: {query}

View on GitHub (pinned to 322425c43b)