{"record":{"id":"7448e178ea9b5cc2","repo":"aaif-goose/goose","slug":"openai-evaluation-failed-str-e","errorCode":null,"errorMessage":"OpenAI evaluation failed: {str(e)}","messagePattern":"OpenAI evaluation failed: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/bench-postprocess-scripts/llm-judges/llm_judge.py","lineNumber":164,"sourceCode":"                            raise ValueError(f\"Failed to parse tie-breaker response after {max_retries} attempts: {str(e)}\")\n                        print(\"Retrying tie-breaker...\")\n                        time.sleep(1)  # Wait 1 second before retrying\n                        continue\n                except Exception as e:\n                    # For other exceptions (API errors, etc.), raise immediately\n                    print(f\"API error in tie-breaker: {str(e)}\")\n                    raise\n        \n        # Get the most common score\n        most_common_score = score_counts.most_common(1)[0][0]\n        print(f\"Most common score: {most_common_score} (occurred {score_counts[most_common_score]} times)\")\n        return most_common_score\n            \n    except Exception as e:\n        if \"OPENAI_API_KEY\" in str(e):\n            raise  # Re-raise API key errors\n        print(f\"Error evaluating with OpenAI: {str(e)}\")\n        raise ValueError(f\"OpenAI evaluation failed: {str(e)}\")\n\n\ndef load_eval_results(working_dir: Path) -> Dict[str, Any]:\n    \"\"\"Load the eval-results.json file from the working directory.\"\"\"\n    eval_results_path = working_dir / \"eval-results.json\"\n    if not eval_results_path.exists():\n        raise FileNotFoundError(f\"eval-results.json not found in {working_dir}\")\n    \n    with open(eval_results_path, 'r') as f:\n        return json.load(f)\n\n\ndef load_output_file(working_dir: Path, output_file: str) -> str:\n    \"\"\"Load the output file to evaluate from the working directory.\"\"\"\n    output_path = working_dir / output_file\n    if not output_path.exists():\n        raise FileNotFoundError(f\"Output file not found: {output_path}\")\n    ","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/scripts/bench-postprocess-scripts/llm-judges/llm_judge.py#L146-L182","documentation":"Raised in the benchmark LLM-judge script as ValueError(f\"OpenAI evaluation failed: {str(e)}\") when any exception escapes the OpenAI-based scoring loop that is not an API-key error (those are re-raised verbatim). It wraps the underlying OpenAI API/SDK failure, so the text after the colon carries the real cause.","triggerScenarios":"Calling the OpenAI judge path of llm_judge.py (evaluate/score flow, including the tie-breaker round) when the OpenAI API call raises anything whose message does not contain 'OPENAI_API_KEY': HTTP 429 rate limit, 5xx, network/timeout, unknown model name, or a response the parser cannot turn into a score.","commonSituations":"Running bench-postprocess scripts without a valid/active OpenAI quota, hitting org rate limits during a long judge run, using a deprecated/renamed judge model, or behind a proxy/firewall that blocks api.openai.com.","solutions":["Read the wrapped text after 'OpenAI evaluation failed:' — it is the underlying API error and names the real cause","If it is 429/quota: wait or lower parallelism/concurrency of judge calls, add exponential backoff","Verify network reachability of api.openai.com (proxy, VPN, firewall) with a curl to /v1/models","Confirm the judge model name still exists on your OpenAI account and the key has credit","If responses are unparseable, tighten the evaluation prompt so the model returns the expected score format"],"exampleFix":"# before\nscore = evaluate_with_openai(output, prompt)\n\n# after\nimport time\nfor attempt in range(3):\n    try:\n        score = evaluate_with_openai(output, prompt)\n        break\n    except ValueError as e:\n        if attempt == 2 or \"OPENAI_API_KEY\" in str(e):\n            raise\n        time.sleep(2 ** attempt)","handlingStrategy":"retry","validationCode":"import os\nassert os.environ.get(\"OPENAI_API_KEY\"), \"OPENAI_API_KEY not set\"\nfrom openai import OpenAI\nOpenAI().models.list()  # cheap pre-flight: key + network + quota","typeGuard":null,"tryCatchPattern":"try:\n    score = evaluate_with_openai(output, prompt)\nexcept ValueError as e:\n    if \"OPENAI_API_KEY\" in str(e):\n        raise  # config problem, do not retry\n    # transient API failure: backoff and retry, else surface wrapped cause\n    log_and_retry(e, attempts=3)","preventionTips":["Run a cheap models.list() pre-flight before long judge batches","Set rate-limit-aware backoff around OpenAI calls","Keep OPENAI_API_KEY in a checked env hook, never inline","Log the wrapped cause string, not just 'evaluation failed'"],"tags":["python","openai","llm","benchmark","api"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}