zed-industries/zed · error · ValueError

unknown benchmark '{benchmark_id}' (valid: {valid})

Error message

unknown benchmark '{benchmark_id}' (valid: {valid})

What it means

get_benchmark() looks up benchmark_id directly in the BENCHMARKS dict and raises when it is not a concrete key, listing the valid ids in the message. Note the asymmetry: aliases (the swe-atlas ones) and groups are only expanded by resolve_benchmarks()/resolve_benchmark_selector(); passing an alias or group straight to get_benchmark fails even though it looks valid.

Source

Thrown at crates/eval_cli/zed_eval/benchmarks.py:175

    "qna": "swe-atlas-qna",
    "rf": "swe-atlas-rf",
    "tw": "swe-atlas-tw",
    "tb21": "terminal-bench-2.1",
}

SWE_ATLAS_PART_BENCHMARKS: dict[str, str] = {
    "qna": "swe-atlas-qna",
    "rf": "swe-atlas-rf",
    "tw": "swe-atlas-tw",
}


def get_benchmark(benchmark_id: str) -> Benchmark:
    try:
        return BENCHMARKS[benchmark_id]
    except KeyError as error:
        valid = ", ".join(sorted(BENCHMARKS))
        raise ValueError(
            f"unknown benchmark '{benchmark_id}' (valid: {valid})"
        ) from error


def is_benchmark_selector(selector: str) -> bool:
    """Whether `selector` names a known benchmark id, alias, or group.

    Used by `run` to validate benchmark positionals before preparing builds."""
    normalized = selector.strip().lower()
    return (
        normalized in BENCHMARK_GROUPS
        or normalized in BENCHMARKS
        or normalized in BENCHMARK_ALIASES
    )


def resolve_benchmark_selector(selector: str) -> list[str]:
    """Expands a user-facing selector (benchmark id, alias, or group) into the

View on GitHub (pinned to bc538def45)

Solutions

1. Use one of the concrete ids listed in the error's (valid: ...) set
2. Expand selectors first: resolve_benchmarks([selector]) maps aliases and groups to concrete ids
3. Normalize input: strip and lowercase before lookup
4. If the id should exist, check BENCHMARKS for a recent rename

Example fix

# before
get_benchmark("qna")   # ValueError: unknown benchmark 'qna' (valid: ...)

# after
from zed_eval.benchmarks import resolve_benchmarks, get_benchmark
benchmark = get_benchmark(resolve_benchmarks(["qna"])[0])   # → swe-atlas-qna

from zed_eval.benchmarks import BENCHMARKS, resolve_benchmarks
ids = resolve_benchmarks(user_selectors)  # expands aliases/groups, raises with the valid set
for benchmark_id in ids:
    assert benchmark_id in BENCHMARKS

from zed_eval.benchmarks import BENCHMARKS

def is_concrete_benchmark_id(benchmark_id):
    return benchmark_id in BENCHMARKS

try:
    benchmark = get_benchmark(benchmark_id)
except ValueError as e:
    print(e)  # message already lists the valid ids
    raise SystemExit(2)

When it happens

Trigger: Calling get_benchmark with an alias ("qna") or group name instead of the concrete id ("swe-atlas-qna"); a typo; mixed case or trailing whitespace in the id.

Common situations: Forwarding user-facing CLI input without resolving selectors first; scripts hardcoding shorthand names; benchmark ids renamed over time.

Related errors

• unknown benchmark '{selector}' (valid: {valid})
• build ids may only contain lowercase letters, numbers, '.',
• could not locate run '{run_id}' in the local run index ({run
• unknown judge preset '{name}' (valid: {valid})
• unknown SWE-Atlas part '{part}' (valid: {valid})

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/f347ab5a4b2ae732. Report an issue: GitHub.