aaif-goose/goose · error · ValueError

OpenAI evaluation failed: {str(e)}

Error message

OpenAI evaluation failed: {str(e)}

What it means

Raised in the benchmark LLM-judge script as ValueError(f"OpenAI evaluation failed: {str(e)}") when any exception escapes the OpenAI-based scoring loop that is not an API-key error (those are re-raised verbatim). It wraps the underlying OpenAI API/SDK failure, so the text after the colon carries the real cause.

Source

Thrown at scripts/bench-postprocess-scripts/llm-judges/llm_judge.py:164

                            raise ValueError(f"Failed to parse tie-breaker response after {max_retries} attempts: {str(e)}")
                        print("Retrying tie-breaker...")
                        time.sleep(1)  # Wait 1 second before retrying
                        continue
                except Exception as e:
                    # For other exceptions (API errors, etc.), raise immediately
                    print(f"API error in tie-breaker: {str(e)}")
                    raise
        
        # Get the most common score
        most_common_score = score_counts.most_common(1)[0][0]
        print(f"Most common score: {most_common_score} (occurred {score_counts[most_common_score]} times)")
        return most_common_score
            
    except Exception as e:
        if "OPENAI_API_KEY" in str(e):
            raise  # Re-raise API key errors
        print(f"Error evaluating with OpenAI: {str(e)}")
        raise ValueError(f"OpenAI evaluation failed: {str(e)}")


def load_eval_results(working_dir: Path) -> Dict[str, Any]:
    """Load the eval-results.json file from the working directory."""
    eval_results_path = working_dir / "eval-results.json"
    if not eval_results_path.exists():
        raise FileNotFoundError(f"eval-results.json not found in {working_dir}")
    
    with open(eval_results_path, 'r') as f:
        return json.load(f)


def load_output_file(working_dir: Path, output_file: str) -> str:
    """Load the output file to evaluate from the working directory."""
    output_path = working_dir / output_file
    if not output_path.exists():
        raise FileNotFoundError(f"Output file not found: {output_path}")
    

View on GitHub (pinned to 3810898a74)

Solutions

1. Read the wrapped text after 'OpenAI evaluation failed:' — it is the underlying API error and names the real cause
2. If it is 429/quota: wait or lower parallelism/concurrency of judge calls, add exponential backoff
3. Verify network reachability of api.openai.com (proxy, VPN, firewall) with a curl to /v1/models
4. Confirm the judge model name still exists on your OpenAI account and the key has credit
5. If responses are unparseable, tighten the evaluation prompt so the model returns the expected score format

Example fix

# before
score = evaluate_with_openai(output, prompt)

# after
import time
for attempt in range(3):
    try:
        score = evaluate_with_openai(output, prompt)
        break
    except ValueError as e:
        if attempt == 2 or "OPENAI_API_KEY" in str(e):
            raise
        time.sleep(2 ** attempt)

import os
assert os.environ.get("OPENAI_API_KEY"), "OPENAI_API_KEY not set"
from openai import OpenAI
OpenAI().models.list()  # cheap pre-flight: key + network + quota

try:
    score = evaluate_with_openai(output, prompt)
except ValueError as e:
    if "OPENAI_API_KEY" in str(e):
        raise  # config problem, do not retry
    # transient API failure: backoff and retry, else surface wrapped cause
    log_and_retry(e, attempts=3)

When it happens

Trigger: Calling the OpenAI judge path of llm_judge.py (evaluate/score flow, including the tie-breaker round) when the OpenAI API call raises anything whose message does not contain 'OPENAI_API_KEY': HTTP 429 rate limit, 5xx, network/timeout, unknown model name, or a response the parser cannot turn into a score.

Common situations: Running bench-postprocess scripts without a valid/active OpenAI quota, hitting org rate limits during a long judge run, using a deprecated/renamed judge model, or behind a proxy/firewall that blocks api.openai.com.

Related errors

• OPENAI_API_KEY environment variable is not set, but is neede
• Failed to parse OpenAI evaluation response after {max_retrie
• eval-results.json not found in {working_dir}
• Output file not found: {output_path}
• Job name must start with a letter or number and contain only

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/7448e178ea9b5cc2. Report an issue: GitHub.