aaif-goose/goose · error · ValueError

Unknown evaluation type: {eval_name}

Error message

Unknown evaluation type: {eval_name}

What it means

Raised by calculate_score in the vibes final-score script when the eval_name argument is neither 'blog_summary' nor 'restaurant_research'. Each eval has its own max-score formula (both current ones divide llm_judge_score + used_fetch_tool + valid_markdown_format by 4.0), so an unknown name has no scoring recipe and the function refuses rather than returning a nonsense score. The name comes from sys.argv[1] in main().

Source

Thrown at scripts/bench-postprocess-scripts/llm-judges/calculate_final_scores_vibes.py:45

    """Calculate the final score based on the evaluation type."""
    llm_judge_score = get_metric_value(metrics, "llm_judge_score")
    used_fetch_tool = get_metric_value(metrics, "used_fetch_tool")
    valid_markdown_format = get_metric_value(metrics, "valid_markdown_format")
    
    if llm_judge_score is None:
        raise ValueError("llm_judge_score not found in metrics")
    
    # Convert boolean metrics to 0/1 if needed
    used_fetch_tool = 1.0 if used_fetch_tool else 0.0
    valid_markdown_format = 1.0 if valid_markdown_format else 0.0
    
    if eval_name == "blog_summary":
        # max score is 4.0 as llm_judge_score is between [0,2] and used_fetch_tool/valid_markedown_format have values [0,1]
        score = (llm_judge_score + used_fetch_tool + valid_markdown_format) / 4.0
    elif eval_name == "restaurant_research":
        score = (llm_judge_score + valid_markdown_format + used_fetch_tool) / 4.0
    else:
        raise ValueError(f"Unknown evaluation type: {eval_name}")
    
    return score


def main():
    if len(sys.argv) != 2:
        print("Usage: calculate_final_score.py <eval_name>")
        sys.exit(1)
    
    eval_name = sys.argv[1]
    
    # Load eval results from current directory
    eval_results_path = Path("eval-results.json")
    if not eval_results_path.exists():
        print(f"Error: eval-results.json not found in current directory")
        sys.exit(1)
    
    with open(eval_results_path, 'r') as f:

View on GitHub (pinned to 3810898a74)

Solutions

1. Pass exactly one of the supported names: blog_summary or restaurant_research
2. Check sys.argv usage: the script takes a single argument, eval_name, and exits with a usage message otherwise
3. When adding a new eval, add an elif branch with its scoring formula before running it

Example fix

# before
python calculate_final_scores_vibes.py blog_summery   # typo

# after
python calculate_final_scores_vibes.py blog_summary   # or restaurant_research

KNOWN_EVALS = {'blog_summary', 'restaurant_research'}
assert eval_name in KNOWN_EVALS, (
    f'eval_name must be one of {sorted(KNOWN_EVALS)}, got {eval_name!r}'
)

When it happens

Trigger: Invoking the script with a typo ('blog_summery'); passing a newer vibes eval name not yet wired into the if/elif; passing the dataset path or model name instead of the eval name.

Common situations: Adding a third vibes eval and forgetting to extend calculate_score; copy-pasting an invocation from older docs with a renamed eval; argument-order mistakes since the script takes exactly one positional argument.

Related errors

• Job name must start with a letter or number and contain only
• --model must be in provider/model form, e.g. anthropic/claud
• --trials must be at least 1
• --concurrency must be at least 1
• --timeout-multiplier must be positive

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/30f10dd4633d03cb. Report an issue: GitHub.