aaif-goose/goose · error · FileNotFoundError

eval-results.json not found in {working_dir}

Error message

eval-results.json not found in {working_dir}

What it means

Python FileNotFoundError raised by load_eval_results() when eval-results.json is absent from the working directory passed to the judge harness. The post-processing script requires the summary file that a previous benchmark/eval run must have written into that directory.

Source

Thrown at scripts/bench-postprocess-scripts/llm-judges/llm_judge.py:171

                    raise
        
        # Get the most common score
        most_common_score = score_counts.most_common(1)[0][0]
        print(f"Most common score: {most_common_score} (occurred {score_counts[most_common_score]} times)")
        return most_common_score
            
    except Exception as e:
        if "OPENAI_API_KEY" in str(e):
            raise  # Re-raise API key errors
        print(f"Error evaluating with OpenAI: {str(e)}")
        raise ValueError(f"OpenAI evaluation failed: {str(e)}")


def load_eval_results(working_dir: Path) -> Dict[str, Any]:
    """Load the eval-results.json file from the working directory."""
    eval_results_path = working_dir / "eval-results.json"
    if not eval_results_path.exists():
        raise FileNotFoundError(f"eval-results.json not found in {working_dir}")
    
    with open(eval_results_path, 'r') as f:
        return json.load(f)


def load_output_file(working_dir: Path, output_file: str) -> str:
    """Load the output file to evaluate from the working directory."""
    output_path = working_dir / output_file
    if not output_path.exists():
        raise FileNotFoundError(f"Output file not found: {output_path}")
    
    with open(output_path, 'r') as f:
        return f.read().strip()


def load_evaluation_prompt(working_dir: Path) -> str:
    """Load the evaluation prompt from a file or use a default.
    

View on GitHub (pinned to 3810898a74)

Solutions

1. Check the directory actually contains eval-results.json: ls <working_dir>/eval-results.json
2. If missing, re-run the benchmark/eval stage that produces it, then re-run the judge
3. Point the script at the directory that holds the eval artifacts (often the run's output folder), not the repo root
4. If the file was renamed or lives one level deeper, copy/rename it to <working_dir>/eval-results.json

Example fix

# before
results = load_eval_results(Path("./out"))

# after
from pathlib import Path
working = Path("./out")
if not (working / "eval-results.json").exists():
    raise SystemExit(f"no eval-results.json in {working}; run the eval stage first")
results = load_eval_results(working)

from pathlib import Path
p = working_dir / "eval-results.json"
if not p.is_file():
    raise SystemExit(f"missing {p}; run the eval stage first")

try:
    results = load_eval_results(working_dir)
except FileNotFoundError as e:
    # fail the pipeline stage with a clear message; do not fabricate results
    raise SystemExit(f"eval artifacts missing in {working_dir}: {e}")

When it happens

Trigger: Invoking the llm-judge post-processing step with a working_dir that never received eval-results.json — e.g. running the judge before the benchmark, pointing at the wrong results directory, or the eval run crashed before writing its summary.

Common situations: Re-running post-processing on a copied/moved results folder where only the model output file was kept; CI jobs that assume a prior stage's artifact exists; typos in the --working-dir style argument.

Related errors

• Output file not found: {output_path}
• --goose-binary does not exist or is not a file: {args.goose_
• OpenAI evaluation failed: {str(e)}
• Update file not found. Please download the update first.
• Job name must start with a letter or number and contain only

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/f3e4d8ed902d8942. Report an issue: GitHub.