What causes "OPENAI_API_KEY environment variable required" in headroomlabs-ai/headroom?

export OPENAI_API_KEY= (a placeholder suffices if the local headroom proxy does not validate it — any non-empty value satisfies this guard)

headroomlabs-ai/headroom · error · ValueError

OPENAI_API_KEY environment variable required

Error message

OPENAI_API_KEY environment variable required

What it means

Run by the headroom-proxy benchmark path: before shelling out to lm-eval against http://localhost:{headroom_port}/v1/chat/completions, it reads OPENAI_API_KEY from the environment and raises ValueError if unset/empty. The key is passed through model_args (api_key=...) so the local-chat-completions client can authenticate — even though the endpoint is the local headroom proxy, the harness still demands a non-empty key string.

Source

Thrown at headroom/evals/comprehensive_benchmark.py:420

    return parse_lm_eval_results(raw_results)


def run_headroom_benchmark(
    model: str = "gpt-4o-mini",
    tasks: list[str] | None = None,
    limit: int | None = None,
    headroom_port: int = 8787,
) -> list[BenchmarkResult]:
    """Run benchmark through Headroom proxy."""
    logger.info(f"Running Headroom benchmark with {model} through proxy...")

    # lm_eval expects base_url to be the full path to chat/completions
    base_url = f"http://localhost:{headroom_port}/v1/chat/completions"

    # Get API key from environment
    api_key = os.environ.get("OPENAI_API_KEY", "")
    if not api_key:
        raise ValueError("OPENAI_API_KEY environment variable required")

    raw_results = run_lm_eval(
        model="local-chat-completions",
        model_args=f"model={model},tokenizer_backend=tiktoken,api_key={api_key}",
        tasks=tasks,
        limit=limit,
        base_url=base_url,
    )

    return parse_lm_eval_results(raw_results)


def compare_results(
    baseline: list[BenchmarkResult],
    headroom: list[BenchmarkResult],
) -> list[ComparisonResult]:
    """Compare baseline and Headroom results."""
    comparisons = []

View on GitHub (pinned to 322425c43b)

Solutions

export OPENAI_API_KEY=<key> (a placeholder suffices if the local headroom proxy does not validate it — any non-empty value satisfies this guard)
For CI, add OPENAI_API_KEY to the job's environment/secret mapping
Confirm with `printenv OPENAI_API_KEY` in the exact shell before invoking the benchmark

Example fix

# before
run_headroom_benchmark(model="gpt-4o", tasks=[...])
# ValueError: OPENAI_API_KEY environment variable required

# after
# export OPENAI_API_KEY=sk-...   (or placeholder for local proxy)
run_headroom_benchmark(model="gpt-4o", tasks=[...])

Defensive patterns

Strategy: validation

Validate before calling

import os

if not os.environ.get("OPENAI_API_KEY"):
    raise SystemExit(
        "OPENAI_API_KEY must be set (any non-empty value if the headroom "
        "proxy does not validate it)"
    )

Prevention

Export OPENAI_API_KEY in every shell/CI job that runs proxy benchmarks
Assert required env vars in a setup step before long benchmark runs
Remember the guard only needs a non-empty string when targeting the local proxy

When it happens

Trigger: Calling the headroom-proxy benchmark runner (run through proxy at headroom_port 8787 by default) in a shell/process where OPENAI_API_KEY was never exported or is set to ''.

Common situations: CI jobs without the secret mapped; shells where the key is set in .env but the file isn't sourced for this run; users assuming a local proxy needs no key at all and skipping it.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/66a252d87cc26720. Report an issue: GitHub.