{"record":{"id":"66a252d87cc26720","repo":"headroomlabs-ai/headroom","slug":"openai-api-key-environment-variable-required","errorCode":null,"errorMessage":"OPENAI_API_KEY environment variable required","messagePattern":"OPENAI_API_KEY environment variable required","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/evals/comprehensive_benchmark.py","lineNumber":420,"sourceCode":"    return parse_lm_eval_results(raw_results)\n\n\ndef run_headroom_benchmark(\n    model: str = \"gpt-4o-mini\",\n    tasks: list[str] | None = None,\n    limit: int | None = None,\n    headroom_port: int = 8787,\n) -> list[BenchmarkResult]:\n    \"\"\"Run benchmark through Headroom proxy.\"\"\"\n    logger.info(f\"Running Headroom benchmark with {model} through proxy...\")\n\n    # lm_eval expects base_url to be the full path to chat/completions\n    base_url = f\"http://localhost:{headroom_port}/v1/chat/completions\"\n\n    # Get API key from environment\n    api_key = os.environ.get(\"OPENAI_API_KEY\", \"\")\n    if not api_key:\n        raise ValueError(\"OPENAI_API_KEY environment variable required\")\n\n    raw_results = run_lm_eval(\n        model=\"local-chat-completions\",\n        model_args=f\"model={model},tokenizer_backend=tiktoken,api_key={api_key}\",\n        tasks=tasks,\n        limit=limit,\n        base_url=base_url,\n    )\n\n    return parse_lm_eval_results(raw_results)\n\n\ndef compare_results(\n    baseline: list[BenchmarkResult],\n    headroom: list[BenchmarkResult],\n) -> list[ComparisonResult]:\n    \"\"\"Compare baseline and Headroom results.\"\"\"\n    comparisons = []","sourceCodeStart":402,"sourceCodeEnd":438,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/evals/comprehensive_benchmark.py#L402-L438","documentation":"Run by the headroom-proxy benchmark path: before shelling out to lm-eval against http://localhost:{headroom_port}/v1/chat/completions, it reads OPENAI_API_KEY from the environment and raises ValueError if unset/empty. The key is passed through model_args (api_key=...) so the local-chat-completions client can authenticate — even though the endpoint is the local headroom proxy, the harness still demands a non-empty key string.","triggerScenarios":"Calling the headroom-proxy benchmark runner (run through proxy at headroom_port 8787 by default) in a shell/process where OPENAI_API_KEY was never exported or is set to ''.","commonSituations":"CI jobs without the secret mapped; shells where the key is set in .env but the file isn't sourced for this run; users assuming a local proxy needs no key at all and skipping it.","solutions":["export OPENAI_API_KEY=<key> (a placeholder suffices if the local headroom proxy does not validate it — any non-empty value satisfies this guard)","For CI, add OPENAI_API_KEY to the job's environment/secret mapping","Confirm with `printenv OPENAI_API_KEY` in the exact shell before invoking the benchmark"],"exampleFix":"# before\nrun_headroom_benchmark(model=\"gpt-4o\", tasks=[...])\n# ValueError: OPENAI_API_KEY environment variable required\n\n# after\n# export OPENAI_API_KEY=sk-...   (or placeholder for local proxy)\nrun_headroom_benchmark(model=\"gpt-4o\", tasks=[...])","handlingStrategy":"validation","validationCode":"import os\n\nif not os.environ.get(\"OPENAI_API_KEY\"):\n    raise SystemExit(\n        \"OPENAI_API_KEY must be set (any non-empty value if the headroom \"\n        \"proxy does not validate it)\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Export OPENAI_API_KEY in every shell/CI job that runs proxy benchmarks","Assert required env vars in a setup step before long benchmark runs","Remember the guard only needs a non-empty string when targeting the local proxy"],"tags":["config","environment","api-key","benchmark","evals"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}