zed-industries/zed · error · ValueError

unknown benchmark '{selector}' (valid: {valid})

Error message

unknown benchmark '{selector}' (valid: {valid})

What it means

resolve_benchmark_selector() expands one user-facing selector into concrete benchmark ids, accepting benchmark ids, group names, and aliases — compared after strip and lowercase. The ValueError fires when the normalized selector matches none of the three registries; the message lists the complete valid set (ids + groups + aliases).

Source

Thrown at crates/eval_cli/zed_eval/benchmarks.py:203

    return (
        normalized in BENCHMARK_GROUPS
        or normalized in BENCHMARKS
        or normalized in BENCHMARK_ALIASES
    )


def resolve_benchmark_selector(selector: str) -> list[str]:
    """Expands a user-facing selector (benchmark id, alias, or group) into the
    concrete benchmark ids it refers to, preserving order."""
    normalized = selector.strip().lower()
    if normalized in BENCHMARK_GROUPS:
        return list(BENCHMARK_GROUPS[normalized])
    if normalized in BENCHMARKS:
        return [normalized]
    if normalized in BENCHMARK_ALIASES:
        return [BENCHMARK_ALIASES[normalized]]
    valid = ", ".join(sorted({*BENCHMARKS, *BENCHMARK_GROUPS, *BENCHMARK_ALIASES}))
    raise ValueError(f"unknown benchmark '{selector}' (valid: {valid})")


def resolve_benchmarks(selectors: list[str]) -> list[str]:
    resolved: list[str] = []
    for selector in selectors:
        for part in selector.split(","):
            part = part.strip()
            if not part:
                continue
            for benchmark_id in resolve_benchmark_selector(part):
                if benchmark_id not in resolved:
                    resolved.append(benchmark_id)
    return resolved


def benchmark_metadata(benchmark: Benchmark) -> dict[str, object]:
    """The self-describing block embedded in a run request so the controller and
    harness-command builder need no separate registry lookup."""

View on GitHub (pinned to bc538def45)

Solutions

1. Pick a name from the (valid: ...) list in the message
2. Prefer resolve_benchmarks(["a,b"]) for comma-separated input — it splits and strips each part before delegating
3. If a documented name stopped resolving, check for a rename and use the new alias

Example fix

# before
resolve_benchmark_selector("QNA,")   # ValueError: unknown benchmark

# after
resolve_benchmarks(["qna,swe-atlas-rf"])   # → ["swe-atlas-qna", "swe-atlas-rf"]

from zed_eval.benchmarks import is_benchmark_selector
if not is_benchmark_selector(selector):
    print(f"unknown benchmark '{selector}'")
    raise SystemExit(2)

from zed_eval.benchmarks import BENCHMARKS, BENCHMARK_GROUPS, BENCHMARK_ALIASES

def is_known_selector(selector):
    n = selector.strip().lower()
    return n in BENCHMARKS or n in BENCHMARK_GROUPS or n in BENCHMARK_ALIASES

try:
    ids = resolve_benchmarks(selectors)
except ValueError as e:
    print(e)  # lists every valid id, group, and alias
    raise SystemExit(2)

When it happens

Trigger: A typo'd or mixed-case selector; an id retired after a rename; calling the function directly with a comma-containing string — splitting on commas is resolve_benchmarks' job, so "a,b" reaches here as one unknown selector.

Common situations: CLI users typing benchmark names from memory; docs referencing an old alias; programmatic callers bypassing resolve_benchmarks' comma handling.

Related errors

• unknown benchmark '{benchmark_id}' (valid: {valid})
• build ids may only contain lowercase letters, numbers, '.',
• could not locate run '{run_id}' in the local run index ({run
• unknown judge preset '{name}' (valid: {valid})
• unknown SWE-Atlas part '{part}' (valid: {valid})

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/bf705669a483c128. Report an issue: GitHub.