{"record":{"id":"f3e4d8ed902d8942","repo":"aaif-goose/goose","slug":"eval-results-json-not-found-in-working-dir","errorCode":null,"errorMessage":"eval-results.json not found in {working_dir}","messagePattern":"eval-results\\.json not found in (.+?)","errorType":"exception","errorClass":"FileNotFoundError","httpStatus":null,"severity":"error","filePath":"scripts/bench-postprocess-scripts/llm-judges/llm_judge.py","lineNumber":171,"sourceCode":"                    raise\n        \n        # Get the most common score\n        most_common_score = score_counts.most_common(1)[0][0]\n        print(f\"Most common score: {most_common_score} (occurred {score_counts[most_common_score]} times)\")\n        return most_common_score\n            \n    except Exception as e:\n        if \"OPENAI_API_KEY\" in str(e):\n            raise  # Re-raise API key errors\n        print(f\"Error evaluating with OpenAI: {str(e)}\")\n        raise ValueError(f\"OpenAI evaluation failed: {str(e)}\")\n\n\ndef load_eval_results(working_dir: Path) -> Dict[str, Any]:\n    \"\"\"Load the eval-results.json file from the working directory.\"\"\"\n    eval_results_path = working_dir / \"eval-results.json\"\n    if not eval_results_path.exists():\n        raise FileNotFoundError(f\"eval-results.json not found in {working_dir}\")\n    \n    with open(eval_results_path, 'r') as f:\n        return json.load(f)\n\n\ndef load_output_file(working_dir: Path, output_file: str) -> str:\n    \"\"\"Load the output file to evaluate from the working directory.\"\"\"\n    output_path = working_dir / output_file\n    if not output_path.exists():\n        raise FileNotFoundError(f\"Output file not found: {output_path}\")\n    \n    with open(output_path, 'r') as f:\n        return f.read().strip()\n\n\ndef load_evaluation_prompt(working_dir: Path) -> str:\n    \"\"\"Load the evaluation prompt from a file or use a default.\n    ","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/scripts/bench-postprocess-scripts/llm-judges/llm_judge.py#L153-L189","documentation":"Python FileNotFoundError raised by load_eval_results() when eval-results.json is absent from the working directory passed to the judge harness. The post-processing script requires the summary file that a previous benchmark/eval run must have written into that directory.","triggerScenarios":"Invoking the llm-judge post-processing step with a working_dir that never received eval-results.json — e.g. running the judge before the benchmark, pointing at the wrong results directory, or the eval run crashed before writing its summary.","commonSituations":"Re-running post-processing on a copied/moved results folder where only the model output file was kept; CI jobs that assume a prior stage's artifact exists; typos in the --working-dir style argument.","solutions":["Check the directory actually contains eval-results.json: ls <working_dir>/eval-results.json","If missing, re-run the benchmark/eval stage that produces it, then re-run the judge","Point the script at the directory that holds the eval artifacts (often the run's output folder), not the repo root","If the file was renamed or lives one level deeper, copy/rename it to <working_dir>/eval-results.json"],"exampleFix":"# before\nresults = load_eval_results(Path(\"./out\"))\n\n# after\nfrom pathlib import Path\nworking = Path(\"./out\")\nif not (working / \"eval-results.json\").exists():\n    raise SystemExit(f\"no eval-results.json in {working}; run the eval stage first\")\nresults = load_eval_results(working)","handlingStrategy":"validation","validationCode":"from pathlib import Path\np = working_dir / \"eval-results.json\"\nif not p.is_file():\n    raise SystemExit(f\"missing {p}; run the eval stage first\")","typeGuard":null,"tryCatchPattern":"try:\n    results = load_eval_results(working_dir)\nexcept FileNotFoundError as e:\n    # fail the pipeline stage with a clear message; do not fabricate results\n    raise SystemExit(f\"eval artifacts missing in {working_dir}: {e}\")","preventionTips":["Make the judge a dependent pipeline stage that runs after eval","Verify artifact existence at stage start, not mid-run","Fail fast instead of generating placeholder results"],"tags":["python","file-not-found","benchmark","filesystem"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}