{"record":{"id":"bf705669a483c128","repo":"zed-industries/zed","slug":"unknown-benchmark-selector-valid-valid","errorCode":null,"errorMessage":"unknown benchmark '{selector}' (valid: {valid})","messagePattern":"unknown benchmark '(.+?)' \\(valid: (.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/benchmarks.py","lineNumber":203,"sourceCode":"    return (\n        normalized in BENCHMARK_GROUPS\n        or normalized in BENCHMARKS\n        or normalized in BENCHMARK_ALIASES\n    )\n\n\ndef resolve_benchmark_selector(selector: str) -> list[str]:\n    \"\"\"Expands a user-facing selector (benchmark id, alias, or group) into the\n    concrete benchmark ids it refers to, preserving order.\"\"\"\n    normalized = selector.strip().lower()\n    if normalized in BENCHMARK_GROUPS:\n        return list(BENCHMARK_GROUPS[normalized])\n    if normalized in BENCHMARKS:\n        return [normalized]\n    if normalized in BENCHMARK_ALIASES:\n        return [BENCHMARK_ALIASES[normalized]]\n    valid = \", \".join(sorted({*BENCHMARKS, *BENCHMARK_GROUPS, *BENCHMARK_ALIASES}))\n    raise ValueError(f\"unknown benchmark '{selector}' (valid: {valid})\")\n\n\ndef resolve_benchmarks(selectors: list[str]) -> list[str]:\n    resolved: list[str] = []\n    for selector in selectors:\n        for part in selector.split(\",\"):\n            part = part.strip()\n            if not part:\n                continue\n            for benchmark_id in resolve_benchmark_selector(part):\n                if benchmark_id not in resolved:\n                    resolved.append(benchmark_id)\n    return resolved\n\n\ndef benchmark_metadata(benchmark: Benchmark) -> dict[str, object]:\n    \"\"\"The self-describing block embedded in a run request so the controller and\n    harness-command builder need no separate registry lookup.\"\"\"","sourceCodeStart":185,"sourceCodeEnd":221,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/benchmarks.py#L185-L221","documentation":"resolve_benchmark_selector() expands one user-facing selector into concrete benchmark ids, accepting benchmark ids, group names, and aliases — compared after strip and lowercase. The ValueError fires when the normalized selector matches none of the three registries; the message lists the complete valid set (ids + groups + aliases).","triggerScenarios":"A typo'd or mixed-case selector; an id retired after a rename; calling the function directly with a comma-containing string — splitting on commas is resolve_benchmarks' job, so \"a,b\" reaches here as one unknown selector.","commonSituations":"CLI users typing benchmark names from memory; docs referencing an old alias; programmatic callers bypassing resolve_benchmarks' comma handling.","solutions":["Pick a name from the (valid: ...) list in the message","Prefer resolve_benchmarks([\"a,b\"]) for comma-separated input — it splits and strips each part before delegating","If a documented name stopped resolving, check for a rename and use the new alias"],"exampleFix":"# before\nresolve_benchmark_selector(\"QNA,\")   # ValueError: unknown benchmark\n\n# after\nresolve_benchmarks([\"qna,swe-atlas-rf\"])   # → [\"swe-atlas-qna\", \"swe-atlas-rf\"]","handlingStrategy":"validation","validationCode":"from zed_eval.benchmarks import is_benchmark_selector\nif not is_benchmark_selector(selector):\n    print(f\"unknown benchmark '{selector}'\")\n    raise SystemExit(2)","typeGuard":"from zed_eval.benchmarks import BENCHMARKS, BENCHMARK_GROUPS, BENCHMARK_ALIASES\n\ndef is_known_selector(selector):\n    n = selector.strip().lower()\n    return n in BENCHMARKS or n in BENCHMARK_GROUPS or n in BENCHMARK_ALIASES","tryCatchPattern":"try:\n    ids = resolve_benchmarks(selectors)\nexcept ValueError as e:\n    print(e)  # lists every valid id, group, and alias\n    raise SystemExit(2)","preventionTips":["Let resolve_benchmarks handle comma lists and whitespace","Validate free-text selectors with is_benchmark_selector before batch jobs","Copy selector names from the error's valid list, not from memory"],"tags":["python","benchmark","validation","cli"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}