zed-industries/zed · error · ValueError

choose at least one benchmark to run

Error message

choose at least one benchmark to run

What it means

Raised by launch.prepare_runs_for_benchmarks() when the resolved benchmark_ids list is empty. This is the shared preparation step behind zed-eval run: it refuses to prepare (or spawn a build for) a launch that would contain zero runs. By the time you reach it, resolve_benchmarks has already expanded ids/aliases/groups, so emptiness means no usable selector was supplied at all.

Source

Thrown at crates/eval_cli/zed_eval/launch.py:557

            build_id=build_id,
            suite_id=suite_id,
        ),
        "benchmark": benchmark_metadata_for_run(args, benchmark),
        "eval_cli_timeout": getattr(args, "eval_cli_timeout", None),
        "extra_env": extra_env,
    }


def prepare_runs_for_benchmarks(
    args: argparse.Namespace,
    benchmark_ids: list[str],
    *,
    suite_id: str | None,
    label_for_benchmark,
    mark_swe_atlas_parts: bool = False,
) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:
    if not benchmark_ids:
        raise ValueError("choose at least one benchmark to run")

    build_id, build_request = prepare_shared_build(args)
    prepared: list[tuple[str, dict[str, Any], dict[str, Any] | None]] = []
    for index, benchmark_id in enumerate(benchmark_ids):
        label = label_for_benchmark(benchmark_id)
        run_request = build_benchmark_run_request(
            args,
            benchmark_id=benchmark_id,
            build_id=build_id,
            suite_id=suite_id,
            index=index,
            run_id_suffix=label,
        )
        if (
            mark_swe_atlas_parts
            and benchmark_id in benchmarks.SWE_ATLAS_PART_BENCHMARKS.values()
        ):
            run_request["suite_part"] = label

View on GitHub (pinned to bc538def45)

Solutions

1. Pass at least one benchmark id, alias, or group: zed-eval run swe-atlas | qna | rf | tw | tb21 | deepswe.
2. In scripts, guard the variable: [ -n "$BENCH" ] || { echo 'no benchmark selected'; exit 1; } before invoking the CLI.
3. Use the group swe-atlas to run all three SWE-Atlas parts in one launch.

Example fix

# before
BENCH=""   # unset/failed substitution
zed-eval run "$BENCH"
# -> ValueError: choose at least one benchmark to run

# after
BENCH="${BENCH:-swe-atlas}"
zed-eval run "$BENCH"

from zed_eval import benchmarks

benchmark_ids = benchmarks.resolve_benchmarks(list(args.benchmark or []))
if not benchmark_ids:
    raise SystemExit("select at least one benchmark id/alias/group (e.g. swe-atlas, qna, tb21)")

def has_selectable_benchmarks(selectors: list[str]) -> bool:
    return any(
        benchmarks.is_benchmark_selector(segment)
        for selector in selectors
        for segment in selector.split(",")
        if segment.strip()
    )

When it happens

Trigger: Running zed-eval run with no benchmark positionals; passing selectors that are only commas/whitespace (e.g. "zed-eval run ,,") which resolve_benchmarks filters to nothing; calling prepare_runs_for_benchmarks programmatically with an empty list.

Common situations: Shell scripts where the benchmark variable expands to empty (unset env var, failed $(...) substitution); quoting mistakes that pass an empty string; argparse setups that make positionals optional.

Related errors

• unknown benchmark '{benchmark_id}' (valid: {valid})
• unknown benchmark '{selector}' (valid: {valid})
• unknown judge preset '{name}' (valid: {valid})
• unknown SWE-Atlas part '{part}' (valid: {valid})
• choose at least one SWE-Atlas part with --parts (e.g. --part

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/91d0f9a6c11bc487. Report an issue: GitHub.