aaif-goose/goose · error · ValueError

llm_judge_score not found in metrics

Error message

llm_judge_score not found in metrics

What it means

Raised by calculate_score in the vibes final-score script when get_metric_value cannot find 'llm_judge_score' in the trial's metrics. get_metric_value only recognizes metric values shaped as a dict with a 'Number' or 'Boolean' key (or returns None otherwise), so the error fires both when the key is absent and when it exists in an unrecognized shape. Every vibes eval treats llm_judge_score as the mandatory core metric; the optional used_fetch_tool and valid_markdown_format metrics default to 0 instead of failing.

Source

Thrown at scripts/bench-postprocess-scripts/llm-judges/calculate_final_scores_vibes.py:33

        if metric[0] == metric_name:
            value = metric[1]
            if "Float" in value:
                return float(value["Float"])
            elif "Integer" in value:
                return float(value["Integer"])
            elif "Boolean" in value:
                return 1.0 if value["Boolean"] else 0.0
    return None


def calculate_score(eval_name, metrics):
    """Calculate the final score based on the evaluation type."""
    llm_judge_score = get_metric_value(metrics, "llm_judge_score")
    used_fetch_tool = get_metric_value(metrics, "used_fetch_tool")
    valid_markdown_format = get_metric_value(metrics, "valid_markdown_format")
    
    if llm_judge_score is None:
        raise ValueError("llm_judge_score not found in metrics")
    
    # Convert boolean metrics to 0/1 if needed
    used_fetch_tool = 1.0 if used_fetch_tool else 0.0
    valid_markdown_format = 1.0 if valid_markdown_format else 0.0
    
    if eval_name == "blog_summary":
        # max score is 4.0 as llm_judge_score is between [0,2] and used_fetch_tool/valid_markedown_format have values [0,1]
        score = (llm_judge_score + used_fetch_tool + valid_markdown_format) / 4.0
    elif eval_name == "restaurant_research":
        score = (llm_judge_score + valid_markdown_format + used_fetch_tool) / 4.0
    else:
        raise ValueError(f"Unknown evaluation type: {eval_name}")
    
    return score


def main():
    if len(sys.argv) != 2:

View on GitHub (pinned to 3810898a74)

Solutions

1. Run the LLM-judge stage for those trials first so metrics include llm_judge_score
2. Open the trial's metrics JSON and confirm the exact key 'llm_judge_score' and its {'Number'|'Boolean': ...} shape
3. If the judge legitimately cannot run for a sample, exclude that sample before final scoring rather than letting it abort the batch
4. If your judge emits a different shape, teach get_metric_value to read it (add the branch) instead of bypassing the check

def has_llm_judge_score(metrics: dict) -> bool:
    return get_metric_value(metrics, 'llm_judge_score') is not None

assert has_llm_judge_score(metrics), f'metrics keys: {sorted(metrics)}'

from typing import Optional

def numeric_judge_score(metrics: dict) -> Optional[float]:
    value = metrics.get('llm_judge_score')
    if isinstance(value, dict):
        if 'Number' in value and isinstance(value['Number'], (int, float)):
            return float(value['Number'])
        if 'Boolean' in value:
            return 1.0 if value['Boolean'] else 0.0
    return None

When it happens

Trigger: Running the final-score pass on trials where the LLM-judge step never executed; the judge wrote its verdict under a different key (judge_score, llm_judge, score); the metric value is a plain float or string rather than the {'Number': x} / {'Boolean': b} envelope get_metric_value expects.

Common situations: Skipping the judge stage to save API cost and then running scoring anyway; a judge-prompt revision that renamed the output metric; metrics JSON produced by a different harness version with a flatter schema.

Related errors

• Unknown evaluation type: {eval_name}
• OPENAI_API_KEY environment variable is not set, but is neede
• Failed to parse OpenAI evaluation response after {max_retrie
• Job name must start with a letter or number and contain only
• --model must be in provider/model form, e.g. anthropic/claud

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/4c23004f09d87e9b. Report an issue: GitHub.