aaif-goose/goose · error · ValueError

OPENAI_API_KEY environment variable is not set, but is neede

Error message

OPENAI_API_KEY environment variable is not set, but is needed to run this evaluation.

What it means

Raised by evaluate_with_openai in the benchmark LLM-judge script before any API call is made. The judge instantiates an OpenAI client to score responses against a rubric, and it reads the key exclusively from the environment via os.getenv('OPENAI_API_KEY'); an unset (or empty) variable aborts with this message. It is a fail-fast guard so runs do not die mid-batch on the first request.

Source

Thrown at scripts/bench-postprocess-scripts/llm-judges/llm_judge.py:52

def evaluate_with_openai(prompt: str, text: str, rubric_max_score: int = 2) -> float:
    """Evaluate response using OpenAI's API.
    
    Args:
        prompt: System prompt for evaluation
        text: Text to evaluate
        rubric_max_score: Maximum score for the rubric (default: 2.0)
        
    Returns:
        float: Evaluation score (0 to rubric_max_score)
        
    Raises:
        ValueError: If OPENAI_API_KEY environment variable is not set
    """
    print("Starting OpenAI evaluation...")
    api_key = os.getenv("OPENAI_API_KEY")
    if not api_key:
        print("No OpenAI API key found!")
        raise ValueError("OPENAI_API_KEY environment variable is not set, but is needed to run this evaluation.")
        
    try:
        client = OpenAI(api_key=api_key)
        
        # Append output instructions to system prompt
        output_instructions = f"""
Output Instructions:
Return your evaluation as a JSON object in the following format:
{{
    "reasoning": "Your brief reasoning for the score",
    "score": <integer between 0 and {rubric_max_score}>
}}

IMPORTANT: 
- Do not use any markdown formatting (no ```json blocks)
- Do not include any additional text before or after the JSON
- Return only the raw JSON object
- The score must be an integer between 0 and {rubric_max_score}"""

View on GitHub (pinned to 3810898a74)

Solutions

Export the variable in the invoking shell before running the judge
For CI, inject OPENAI_API_KEY as a masked secret into the job environment
Confirm it is set and non-empty without printing it: python -c "import os; assert os.getenv('OPENAI_API_KEY')"

Example fix

# before
# key never exported; judge aborts on first call
python llm_judge.py ...

# after
export OPENAI_API_KEY='<your-key-here>'
python llm_judge.py ...

Defensive patterns

Strategy: validation

Validate before calling

import os

assert os.getenv('OPENAI_API_KEY'), (
    'export OPENAI_API_KEY (non-empty) before running the judge'
)

Prevention

Export the key in the invoking shell or inject it as a masked CI secret
Note this script reads only the environment — it does not load .env files itself
Verify presence without printing the value: python -c "import os; assert os.getenv('OPENAI_API_KEY')"

When it happens

Trigger: Running the judge in a fresh shell or CI container without exporting the key; the key present in a .env file this script never loads; the variable name misspelled (OPENAI_APIKEY, OPENAI_KEY); an exported empty value.

Common situations: CI jobs with clean environments; local runs where the developer authenticated via a wrapper tool but never exported the variable; rotating keys and the shell still holding nothing.

Related errors

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/0baffcad0e83eca0. Report an issue: GitHub.