{"record":{"id":"4c23004f09d87e9b","repo":"aaif-goose/goose","slug":"llm-judge-score-not-found-in-metrics","errorCode":null,"errorMessage":"llm_judge_score not found in metrics","messagePattern":"llm_judge_score not found in metrics","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/bench-postprocess-scripts/llm-judges/calculate_final_scores_vibes.py","lineNumber":33,"sourceCode":"        if metric[0] == metric_name:\n            value = metric[1]\n            if \"Float\" in value:\n                return float(value[\"Float\"])\n            elif \"Integer\" in value:\n                return float(value[\"Integer\"])\n            elif \"Boolean\" in value:\n                return 1.0 if value[\"Boolean\"] else 0.0\n    return None\n\n\ndef calculate_score(eval_name, metrics):\n    \"\"\"Calculate the final score based on the evaluation type.\"\"\"\n    llm_judge_score = get_metric_value(metrics, \"llm_judge_score\")\n    used_fetch_tool = get_metric_value(metrics, \"used_fetch_tool\")\n    valid_markdown_format = get_metric_value(metrics, \"valid_markdown_format\")\n    \n    if llm_judge_score is None:\n        raise ValueError(\"llm_judge_score not found in metrics\")\n    \n    # Convert boolean metrics to 0/1 if needed\n    used_fetch_tool = 1.0 if used_fetch_tool else 0.0\n    valid_markdown_format = 1.0 if valid_markdown_format else 0.0\n    \n    if eval_name == \"blog_summary\":\n        # max score is 4.0 as llm_judge_score is between [0,2] and used_fetch_tool/valid_markedown_format have values [0,1]\n        score = (llm_judge_score + used_fetch_tool + valid_markdown_format) / 4.0\n    elif eval_name == \"restaurant_research\":\n        score = (llm_judge_score + valid_markdown_format + used_fetch_tool) / 4.0\n    else:\n        raise ValueError(f\"Unknown evaluation type: {eval_name}\")\n    \n    return score\n\n\ndef main():\n    if len(sys.argv) != 2:","sourceCodeStart":15,"sourceCodeEnd":51,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/scripts/bench-postprocess-scripts/llm-judges/calculate_final_scores_vibes.py#L15-L51","documentation":"Raised by calculate_score in the vibes final-score script when get_metric_value cannot find 'llm_judge_score' in the trial's metrics. get_metric_value only recognizes metric values shaped as a dict with a 'Number' or 'Boolean' key (or returns None otherwise), so the error fires both when the key is absent and when it exists in an unrecognized shape. Every vibes eval treats llm_judge_score as the mandatory core metric; the optional used_fetch_tool and valid_markdown_format metrics default to 0 instead of failing.","triggerScenarios":"Running the final-score pass on trials where the LLM-judge step never executed; the judge wrote its verdict under a different key (judge_score, llm_judge, score); the metric value is a plain float or string rather than the {'Number': x} / {'Boolean': b} envelope get_metric_value expects.","commonSituations":"Skipping the judge stage to save API cost and then running scoring anyway; a judge-prompt revision that renamed the output metric; metrics JSON produced by a different harness version with a flatter schema.","solutions":["Run the LLM-judge stage for those trials first so metrics include llm_judge_score","Open the trial's metrics JSON and confirm the exact key 'llm_judge_score' and its {'Number'|'Boolean': ...} shape","If the judge legitimately cannot run for a sample, exclude that sample before final scoring rather than letting it abort the batch","If your judge emits a different shape, teach get_metric_value to read it (add the branch) instead of bypassing the check"],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"def has_llm_judge_score(metrics: dict) -> bool:\n    return get_metric_value(metrics, 'llm_judge_score') is not None\n\nassert has_llm_judge_score(metrics), f'metrics keys: {sorted(metrics)}'","typeGuard":"from typing import Optional\n\ndef numeric_judge_score(metrics: dict) -> Optional[float]:\n    value = metrics.get('llm_judge_score')\n    if isinstance(value, dict):\n        if 'Number' in value and isinstance(value['Number'], (int, float)):\n            return float(value['Number'])\n        if 'Boolean' in value:\n            return 1.0 if value['Boolean'] else 0.0\n    return None","tryCatchPattern":null,"preventionTips":["Run the judge stage before the final-score pass","Standardize the metric key 'llm_judge_score' and its {'Number'|'Boolean': ...} envelope across judge prompts","Filter out un-judged samples before batch scoring instead of letting one bad sample abort the run"],"tags":["metrics","llm-judge","benchmark","scoring"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}