{"record":{"id":"91d0f9a6c11bc487","repo":"zed-industries/zed","slug":"choose-at-least-one-benchmark-to-run","errorCode":null,"errorMessage":"choose at least one benchmark to run","messagePattern":"choose at least one benchmark to run","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/launch.py","lineNumber":557,"sourceCode":"            build_id=build_id,\n            suite_id=suite_id,\n        ),\n        \"benchmark\": benchmark_metadata_for_run(args, benchmark),\n        \"eval_cli_timeout\": getattr(args, \"eval_cli_timeout\", None),\n        \"extra_env\": extra_env,\n    }\n\n\ndef prepare_runs_for_benchmarks(\n    args: argparse.Namespace,\n    benchmark_ids: list[str],\n    *,\n    suite_id: str | None,\n    label_for_benchmark,\n    mark_swe_atlas_parts: bool = False,\n) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:\n    if not benchmark_ids:\n        raise ValueError(\"choose at least one benchmark to run\")\n\n    build_id, build_request = prepare_shared_build(args)\n    prepared: list[tuple[str, dict[str, Any], dict[str, Any] | None]] = []\n    for index, benchmark_id in enumerate(benchmark_ids):\n        label = label_for_benchmark(benchmark_id)\n        run_request = build_benchmark_run_request(\n            args,\n            benchmark_id=benchmark_id,\n            build_id=build_id,\n            suite_id=suite_id,\n            index=index,\n            run_id_suffix=label,\n        )\n        if (\n            mark_swe_atlas_parts\n            and benchmark_id in benchmarks.SWE_ATLAS_PART_BENCHMARKS.values()\n        ):\n            run_request[\"suite_part\"] = label","sourceCodeStart":539,"sourceCodeEnd":575,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/launch.py#L539-L575","documentation":"Raised by launch.prepare_runs_for_benchmarks() when the resolved benchmark_ids list is empty. This is the shared preparation step behind zed-eval run: it refuses to prepare (or spawn a build for) a launch that would contain zero runs. By the time you reach it, resolve_benchmarks has already expanded ids/aliases/groups, so emptiness means no usable selector was supplied at all.","triggerScenarios":"Running zed-eval run with no benchmark positionals; passing selectors that are only commas/whitespace (e.g. \"zed-eval run ,,\") which resolve_benchmarks filters to nothing; calling prepare_runs_for_benchmarks programmatically with an empty list.","commonSituations":"Shell scripts where the benchmark variable expands to empty (unset env var, failed $(...) substitution); quoting mistakes that pass an empty string; argparse setups that make positionals optional.","solutions":["Pass at least one benchmark id, alias, or group: zed-eval run swe-atlas | qna | rf | tw | tb21 | deepswe.","In scripts, guard the variable: [ -n \"$BENCH\" ] || { echo 'no benchmark selected'; exit 1; } before invoking the CLI.","Use the group swe-atlas to run all three SWE-Atlas parts in one launch."],"exampleFix":"# before\nBENCH=\"\"   # unset/failed substitution\nzed-eval run \"$BENCH\"\n# -> ValueError: choose at least one benchmark to run\n\n# after\nBENCH=\"${BENCH:-swe-atlas}\"\nzed-eval run \"$BENCH\"","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\nbenchmark_ids = benchmarks.resolve_benchmarks(list(args.benchmark or []))\nif not benchmark_ids:\n    raise SystemExit(\"select at least one benchmark id/alias/group (e.g. swe-atlas, qna, tb21)\")","typeGuard":"def has_selectable_benchmarks(selectors: list[str]) -> bool:\n    return any(\n        benchmarks.is_benchmark_selector(segment)\n        for selector in selectors\n        for segment in selector.split(\",\")\n        if segment.strip()\n    )","tryCatchPattern":null,"preventionTips":["Default empty script variables before invoking the CLI: BENCH=\"${BENCH:-swe-atlas}\".","Run zed-eval list / check benchmarks.py for valid ids, aliases, and groups.","Avoid passing bare commas or whitespace-only selector strings."],"tags":["cli","validation","launch"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}