{"record":{"id":"09becf423a4ba2cb","repo":"abhigyanpatwari/GitNexus","slug":"unsupported-promotion-metric-metric","errorCode":null,"errorMessage":"unsupported promotion metric: {metric}","messagePattern":"unsupported promotion metric: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"eval/workflow_bench/evolution.py","lineNumber":496,"sourceCode":"def evaluate_candidate(\n    results: dict[str, dict[str, dict[str, Any]]],\n    *,\n    incumbent_arm: str,\n    candidate_arm: str,\n    model: str | None,\n    metric: str = \"cost_usd\",\n    min_runs: int = 3,\n    min_improvement_pct: float = 5.0,\n    max_task_regression_pct: float = 20.0,\n) -> dict[str, Any]:\n    \"\"\"Deterministically decide whether a prompt candidate is promotable.\n\n    Resolution is lexicographically primary: a cheaper candidate that fails\n    more tasks never wins. With equal quality, the candidate must clear the\n    configured median efficiency gain without a large per-task regression.\n    \"\"\"\n    if metric not in PROMOTION_METRICS:\n        raise ValueError(f\"unsupported promotion metric: {metric}\")\n\n    reasons: list[str] = []\n    task_rows: list[dict[str, Any]] = []\n    insufficient = False\n    quality_regression = False\n    quality_floor_failed = False\n    efficiency_regression = False\n\n    if not model:\n        insufficient = True\n        reasons.append(\"a named --model is required so prompt evidence cannot drift\")\n\n    for task_id, arms in sorted(results.items()):\n        if incumbent_arm not in arms or candidate_arm not in arms:\n            insufficient = True\n            reasons.append(f\"{task_id}: both {incumbent_arm} and {candidate_arm} are required\")\n            continue\n","sourceCodeStart":478,"sourceCodeEnd":514,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/evolution.py#L478-L514","documentation":"evaluate_candidate compares candidate vs incumbent on exactly one efficiency axis, so the metric string must be in PROMOTION_METRICS = ('output_tokens','cost_usd','duration_s','num_turns'). Any other value is rejected before any data is read, because an unknown metric would make the lexicographic quality-then-efficiency decision meaningless.","triggerScenarios":"Caller passes metric='tokens' (the real name is 'output_tokens'), 'time' (real name 'duration_s'), 'latency_ms', 'usd', or a free-form string from a CLI flag.","commonSituations":"A CLI wired to a free-text --metric; a caller from before a metric was renamed; copy-paste from docs that used a shorthand.","solutions":["Use one of the four supported names exactly: output_tokens, cost_usd, duration_s, num_turns.","If you genuinely need a new axis, add it to PROMOTION_METRICS in evolution.py and wire its aggregation into the results rollup that feeds evaluate_candidate.","Pin the CLI choices to the tuple so typos fail at argparse time."],"exampleFix":"# before\nevaluate_candidate(results, incumbent_arm=\"a\", candidate_arm=\"b\", model=\"sonnet\", metric=\"tokens\")\n\n# after\nevaluate_candidate(results, incumbent_arm=\"a\", candidate_arm=\"b\", model=\"sonnet\", metric=\"output_tokens\")\n\n# and pin the CLI source of truth\nimport argparse\nfrom eval.workflow_bench.evolution import PROMOTION_METRICS\nparser.add_argument(\"--metric\", choices=PROMOTION_METRICS, default=\"cost_usd\")","handlingStrategy":"validation","validationCode":"from eval.workflow_bench.evolution import PROMOTION_METRICS\n\ndef validate_metric(metric: str) -> str:\n    if metric not in PROMOTION_METRICS:\n        raise ValueError(f\"metric must be one of {PROMOTION_METRICS}, got {metric!r}\")\n    return metric","typeGuard":"from eval.workflow_bench.evolution import PROMOTION_METRICS\nfrom typing import Literal\n\nSupportedMetric = Literal[\"output_tokens\", \"cost_usd\", \"duration_s\", \"num_turns\"]\n\ndef is_supported_metric(value: object) -> TypeGuard[SupportedMetric]:\n    return value in PROMOTION_METRICS","tryCatchPattern":null,"preventionTips":["Pin CLI --metric to choices=PROMOTION_METRICS so typos fail at argparse.","Reference the tuple, do not hard-code the four names at call sites.","When adding a metric, update PROMOTION_METRICS and the results rollup together."],"tags":["workflow-bench","promotion","validation","api-contract"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}