headroomlabs-ai/headroom · error · ValueError

OpenAI API key required. Set OPENAI_API_KEY environment vari

Error message

OpenAI API key required. Set OPENAI_API_KEY environment variable or pass api_key parameter.

What it means

Raised by the LLM-judge prompt comparison in headroom.evals.prompt_comparison when no OpenAI API key can be resolved. The semantic-equivalence judge needs a live OpenAI call, so the function checks the explicit api_key parameter first, then the OPENAI_API_KEY environment variable, and refuses to proceed if both are empty. It is a configuration error thrown before any network traffic happens.

Source

Thrown at headroom/evals/prompt_comparison.py:257

    Example:
        result = compare_prompts(
            original_prompt="Explain quantum computing in simple terms.",
            headroom_modified_prompt="Explain quantum computing in simple terms.",
        )
        if not result.are_equivalent:
            print(f"WARNING: Prompts differ! {result.differences}")
    """
    try:
        from openai import OpenAI
    except ImportError as e:
        raise ImportError(
            "OpenAI package required for prompt comparison. Install with: pip install openai"
        ) from e

    # Get API key
    resolved_api_key = api_key or os.environ.get("OPENAI_API_KEY")
    if not resolved_api_key:
        raise ValueError(
            "OpenAI API key required. Set OPENAI_API_KEY environment variable "
            "or pass api_key parameter."
        )

    client = OpenAI(api_key=resolved_api_key)

    # Build the judge prompt
    judge_prompt = SEMANTIC_EQUIVALENCE_JUDGE_PROMPT.format(
        original_prompt=original_prompt,
        modified_prompt=headroom_modified_prompt,
    )

    # Call the judge
    response = client.chat.completions.create(
        model=judge_model,
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0.0,  # Deterministic for consistent evaluation
        max_tokens=500,

View on GitHub (pinned to 322425c43b)

Solutions

  1. Export the key: export OPENAI_API_KEY=sk-... in the shell (or add it to your .env / CI secrets) and rerun.
  2. Pass the key explicitly: compare_messages(..., api_key='sk-...') so the code does not depend on the environment.
  3. If the judge model is configurable, point judge_model at a provider that does not need OpenAI credentials.
  4. Verify with: python -c "import os; print(bool(os.environ.get('OPENAI_API_KEY')))" before launching the eval.

Example fix

# before
result = compare_messages(original_messages, modified_messages)  # ValueError: no key

# after
result = compare_messages(
    original_messages,
    modified_messages,
    api_key=os.environ["OPENAI_API_KEY"],  # explicit, fails loudly at your boundary
)
Defensive patterns

Strategy: validation

Validate before calling

import os

api_key = os.environ.get("OPENAI_API_KEY")
if not api_key:
    raise SystemExit("OPENAI_API_KEY not set — the prompt-comparison judge requires it")

Try / catch

try:
    result = compare_messages(original_messages, modified_messages, api_key=api_key)
except ValueError as e:
    if "API key" in str(e):
        logger.error("judge not configured: %s", e); sys.exit(2)
    raise

Prevention

When it happens

Trigger: Calling compare_messages()/SEMANTIC_EQUIVALENCE judge entry point (prompt_comparison.py:257 region) with api_key=None/omitted while OPENAI_API_KEY is unset or empty in the process environment (e.g. fresh shell, CI runner, container without the env var, or env var stripped by a subprocess wrapper).

Common situations: Running the prompt-preservation eval suite locally for the first time; CI pipelines that do not export OPENAI_API_KEY; dotenv files loaded after import time; running under a different user/service account than the one holding the key.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/770f08ab197475ff. Report an issue: GitHub.