{"record":{"id":"f347ab5a4b2ae732","repo":"zed-industries/zed","slug":"unknown-benchmark-benchmark-id-valid-valid","errorCode":null,"errorMessage":"unknown benchmark '{benchmark_id}' (valid: {valid})","messagePattern":"unknown benchmark '(.+?)' \\(valid: (.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/benchmarks.py","lineNumber":175,"sourceCode":"    \"qna\": \"swe-atlas-qna\",\n    \"rf\": \"swe-atlas-rf\",\n    \"tw\": \"swe-atlas-tw\",\n    \"tb21\": \"terminal-bench-2.1\",\n}\n\nSWE_ATLAS_PART_BENCHMARKS: dict[str, str] = {\n    \"qna\": \"swe-atlas-qna\",\n    \"rf\": \"swe-atlas-rf\",\n    \"tw\": \"swe-atlas-tw\",\n}\n\n\ndef get_benchmark(benchmark_id: str) -> Benchmark:\n    try:\n        return BENCHMARKS[benchmark_id]\n    except KeyError as error:\n        valid = \", \".join(sorted(BENCHMARKS))\n        raise ValueError(\n            f\"unknown benchmark '{benchmark_id}' (valid: {valid})\"\n        ) from error\n\n\ndef is_benchmark_selector(selector: str) -> bool:\n    \"\"\"Whether `selector` names a known benchmark id, alias, or group.\n\n    Used by `run` to validate benchmark positionals before preparing builds.\"\"\"\n    normalized = selector.strip().lower()\n    return (\n        normalized in BENCHMARK_GROUPS\n        or normalized in BENCHMARKS\n        or normalized in BENCHMARK_ALIASES\n    )\n\n\ndef resolve_benchmark_selector(selector: str) -> list[str]:\n    \"\"\"Expands a user-facing selector (benchmark id, alias, or group) into the","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/benchmarks.py#L157-L193","documentation":"get_benchmark() looks up benchmark_id directly in the BENCHMARKS dict and raises when it is not a concrete key, listing the valid ids in the message. Note the asymmetry: aliases (the swe-atlas ones) and groups are only expanded by resolve_benchmarks()/resolve_benchmark_selector(); passing an alias or group straight to get_benchmark fails even though it looks valid.","triggerScenarios":"Calling get_benchmark with an alias (\"qna\") or group name instead of the concrete id (\"swe-atlas-qna\"); a typo; mixed case or trailing whitespace in the id.","commonSituations":"Forwarding user-facing CLI input without resolving selectors first; scripts hardcoding shorthand names; benchmark ids renamed over time.","solutions":["Use one of the concrete ids listed in the error's (valid: ...) set","Expand selectors first: resolve_benchmarks([selector]) maps aliases and groups to concrete ids","Normalize input: strip and lowercase before lookup","If the id should exist, check BENCHMARKS for a recent rename"],"exampleFix":"# before\nget_benchmark(\"qna\")   # ValueError: unknown benchmark 'qna' (valid: ...)\n\n# after\nfrom zed_eval.benchmarks import resolve_benchmarks, get_benchmark\nbenchmark = get_benchmark(resolve_benchmarks([\"qna\"])[0])   # → swe-atlas-qna","handlingStrategy":"validation","validationCode":"from zed_eval.benchmarks import BENCHMARKS, resolve_benchmarks\nids = resolve_benchmarks(user_selectors)  # expands aliases/groups, raises with the valid set\nfor benchmark_id in ids:\n    assert benchmark_id in BENCHMARKS","typeGuard":"from zed_eval.benchmarks import BENCHMARKS\n\ndef is_concrete_benchmark_id(benchmark_id):\n    return benchmark_id in BENCHMARKS","tryCatchPattern":"try:\n    benchmark = get_benchmark(benchmark_id)\nexcept ValueError as e:\n    print(e)  # message already lists the valid ids\n    raise SystemExit(2)","preventionTips":["Route all external input through resolve_benchmarks before get_benchmark","Never hand-map aliases yourself — rely on BENCHMARK_ALIASES","Assert membership in BENCHMARKS for ids hardcoded in tests"],"tags":["python","benchmark","validation","cli"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}