abhigyanpatwari/GitNexus · error · ValueError

unsupported promotion metric: {metric}

Error message

unsupported promotion metric: {metric}

What it means

evaluate_candidate compares candidate vs incumbent on exactly one efficiency axis, so the metric string must be in PROMOTION_METRICS = ('output_tokens','cost_usd','duration_s','num_turns'). Any other value is rejected before any data is read, because an unknown metric would make the lexicographic quality-then-efficiency decision meaningless.

Source

Thrown at eval/workflow_bench/evolution.py:496

def evaluate_candidate(
    results: dict[str, dict[str, dict[str, Any]]],
    *,
    incumbent_arm: str,
    candidate_arm: str,
    model: str | None,
    metric: str = "cost_usd",
    min_runs: int = 3,
    min_improvement_pct: float = 5.0,
    max_task_regression_pct: float = 20.0,
) -> dict[str, Any]:
    """Deterministically decide whether a prompt candidate is promotable.

    Resolution is lexicographically primary: a cheaper candidate that fails
    more tasks never wins. With equal quality, the candidate must clear the
    configured median efficiency gain without a large per-task regression.
    """
    if metric not in PROMOTION_METRICS:
        raise ValueError(f"unsupported promotion metric: {metric}")

    reasons: list[str] = []
    task_rows: list[dict[str, Any]] = []
    insufficient = False
    quality_regression = False
    quality_floor_failed = False
    efficiency_regression = False

    if not model:
        insufficient = True
        reasons.append("a named --model is required so prompt evidence cannot drift")

    for task_id, arms in sorted(results.items()):
        if incumbent_arm not in arms or candidate_arm not in arms:
            insufficient = True
            reasons.append(f"{task_id}: both {incumbent_arm} and {candidate_arm} are required")
            continue

View on GitHub (pinned to d540b00184)

Solutions

1. Use one of the four supported names exactly: output_tokens, cost_usd, duration_s, num_turns.
2. If you genuinely need a new axis, add it to PROMOTION_METRICS in evolution.py and wire its aggregation into the results rollup that feeds evaluate_candidate.
3. Pin the CLI choices to the tuple so typos fail at argparse time.

Example fix

# before
evaluate_candidate(results, incumbent_arm="a", candidate_arm="b", model="sonnet", metric="tokens")

# after
evaluate_candidate(results, incumbent_arm="a", candidate_arm="b", model="sonnet", metric="output_tokens")

# and pin the CLI source of truth
import argparse
from eval.workflow_bench.evolution import PROMOTION_METRICS
parser.add_argument("--metric", choices=PROMOTION_METRICS, default="cost_usd")

from eval.workflow_bench.evolution import PROMOTION_METRICS

def validate_metric(metric: str) -> str:
    if metric not in PROMOTION_METRICS:
        raise ValueError(f"metric must be one of {PROMOTION_METRICS}, got {metric!r}")
    return metric

from eval.workflow_bench.evolution import PROMOTION_METRICS
from typing import Literal

SupportedMetric = Literal["output_tokens", "cost_usd", "duration_s", "num_turns"]

def is_supported_metric(value: object) -> TypeGuard[SupportedMetric]:
    return value in PROMOTION_METRICS

When it happens

Trigger: Caller passes metric='tokens' (the real name is 'output_tokens'), 'time' (real name 'duration_s'), 'latency_ms', 'usd', or a free-form string from a CLI flag.

Common situations: A CLI wired to a free-text --metric; a caller from before a metric was renamed; copy-paste from docs that used a shorthand.

Related errors

• promotion binding uses an unsupported schema
• promotion binding has no selected tasks
• promotion binding is missing hidden-oracle or dependency dig
• promotion binding is missing hidden-oracle files
• {label} is unavailable: {path}: {exc}

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/09becf423a4ba2cb. Report an issue: GitHub.