{"record":{"id":"30f10dd4633d03cb","repo":"aaif-goose/goose","slug":"unknown-evaluation-type-eval-name","errorCode":null,"errorMessage":"Unknown evaluation type: {eval_name}","messagePattern":"Unknown evaluation type: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/bench-postprocess-scripts/llm-judges/calculate_final_scores_vibes.py","lineNumber":45,"sourceCode":"    \"\"\"Calculate the final score based on the evaluation type.\"\"\"\n    llm_judge_score = get_metric_value(metrics, \"llm_judge_score\")\n    used_fetch_tool = get_metric_value(metrics, \"used_fetch_tool\")\n    valid_markdown_format = get_metric_value(metrics, \"valid_markdown_format\")\n    \n    if llm_judge_score is None:\n        raise ValueError(\"llm_judge_score not found in metrics\")\n    \n    # Convert boolean metrics to 0/1 if needed\n    used_fetch_tool = 1.0 if used_fetch_tool else 0.0\n    valid_markdown_format = 1.0 if valid_markdown_format else 0.0\n    \n    if eval_name == \"blog_summary\":\n        # max score is 4.0 as llm_judge_score is between [0,2] and used_fetch_tool/valid_markedown_format have values [0,1]\n        score = (llm_judge_score + used_fetch_tool + valid_markdown_format) / 4.0\n    elif eval_name == \"restaurant_research\":\n        score = (llm_judge_score + valid_markdown_format + used_fetch_tool) / 4.0\n    else:\n        raise ValueError(f\"Unknown evaluation type: {eval_name}\")\n    \n    return score\n\n\ndef main():\n    if len(sys.argv) != 2:\n        print(\"Usage: calculate_final_score.py <eval_name>\")\n        sys.exit(1)\n    \n    eval_name = sys.argv[1]\n    \n    # Load eval results from current directory\n    eval_results_path = Path(\"eval-results.json\")\n    if not eval_results_path.exists():\n        print(f\"Error: eval-results.json not found in current directory\")\n        sys.exit(1)\n    \n    with open(eval_results_path, 'r') as f:","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/scripts/bench-postprocess-scripts/llm-judges/calculate_final_scores_vibes.py#L27-L63","documentation":"Raised by calculate_score in the vibes final-score script when the eval_name argument is neither 'blog_summary' nor 'restaurant_research'. Each eval has its own max-score formula (both current ones divide llm_judge_score + used_fetch_tool + valid_markdown_format by 4.0), so an unknown name has no scoring recipe and the function refuses rather than returning a nonsense score. The name comes from sys.argv[1] in main().","triggerScenarios":"Invoking the script with a typo ('blog_summery'); passing a newer vibes eval name not yet wired into the if/elif; passing the dataset path or model name instead of the eval name.","commonSituations":"Adding a third vibes eval and forgetting to extend calculate_score; copy-pasting an invocation from older docs with a renamed eval; argument-order mistakes since the script takes exactly one positional argument.","solutions":["Pass exactly one of the supported names: blog_summary or restaurant_research","Check sys.argv usage: the script takes a single argument, eval_name, and exits with a usage message otherwise","When adding a new eval, add an elif branch with its scoring formula before running it"],"exampleFix":"# before\npython calculate_final_scores_vibes.py blog_summery   # typo\n\n# after\npython calculate_final_scores_vibes.py blog_summary   # or restaurant_research","handlingStrategy":"validation","validationCode":"KNOWN_EVALS = {'blog_summary', 'restaurant_research'}\nassert eval_name in KNOWN_EVALS, (\n    f'eval_name must be one of {sorted(KNOWN_EVALS)}, got {eval_name!r}'\n)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pass exactly one positional argument: the eval name","Add new evals to the if/elif (with their max-score formula) before invoking the script with them","Validate the name in CI pipelines that loop over eval lists"],"tags":["cli","validation","llm-judge","benchmark","scoring"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}