zed-industries/zed · error · ValueError

run request is missing a 'benchmark' block

Error message

run request is missing a 'benchmark' block

What it means

Raised by harness_command._benchmark_block() when run_request.get("benchmark") is missing, None, or not a dict. Every run request produced by launch.build_benchmark_run_request embeds a self-describing benchmark block from benchmarks.benchmark_metadata(); both build_harness_command() and run_metadata() (called by the controller's write_run_inputs) require it so the harness command can be built without a separate registry lookup.

Source

Thrown at crates/eval_cli/zed_eval/harness_command.py:19

"""Build the harness command (Harbor or Pier) for a benchmark run request.

The command is driven by the self-describing `benchmark` block embedded in a run
request (see `benchmarks.benchmark_metadata`), so it works for any registered
benchmark without a separate experiment registry lookup.
"""

from __future__ import annotations

import json
from typing import Any

from . import benchmarks, config


def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:
    block = run_request.get("benchmark")
    if not isinstance(block, dict):
        raise ValueError("run request is missing a 'benchmark' block")
    return block


def dataset_args(benchmark: dict[str, Any]) -> list[str]:
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind == benchmarks.DATASET_REGISTRY:
        name = dataset.get("name")
        if not name:
            raise ValueError("registry dataset requires a name")
        return ["-d", name]
    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        data_dir = dataset.get("data_dir")
        if not data_dir:
            raise ValueError("path dataset requires data_dir")
        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")

View on GitHub (pinned to bc538def45)

Solutions

1. Embed the registry block before dispatching: run_request["benchmark"] = benchmarks.benchmark_metadata(benchmarks.get_benchmark(benchmark_id)).
2. If replaying a stored request, use the untouched /data/runs/.../request.json rather than a hand-edited copy.
3. Add an assertion before spawning the controller: assert isinstance(run_request.get("benchmark"), dict).

Example fix

# before
run_request = {
    "run_id": "r1",
    "namespace": "me",
    # no "benchmark" block -> ValueError in build_harness_command
}

# after
from zed_eval import benchmarks
run_request["benchmark"] = benchmarks.benchmark_metadata(
    benchmarks.get_benchmark("swe-atlas-rf")
)

from zed_eval import benchmarks

if not isinstance(run_request.get("benchmark"), dict):
    run_request["benchmark"] = benchmarks.benchmark_metadata(
        benchmarks.get_benchmark(benchmark_id)
    )

from typing import Any

def has_benchmark_block(run_request: dict[str, Any]) -> bool:
    """True when the run request carries a usable self-describing benchmark block."""
    return isinstance(run_request.get("benchmark"), dict)

try:
    command = harness_command.build_harness_command(run_request, jobs_dir)
except ValueError as error:
    raise SystemExit(f"run request rejected: {error}")

When it happens

Trigger: Calling harness_command.build_harness_command(run_request, jobs_dir) or harness_command.run_metadata(run_request) with a hand-constructed dict that lacks the "benchmark" key; loading a saved request.json from the volume and mutating/dropping fields before replaying it; a rejudge-style request (which carries no benchmark block) accidentally fed to the harness-command builder.

Common situations: Scripting the controller directly instead of going through zed-eval run; partial-JSON round-trips where the block was serialized under a different key; code written against an older request schema that kept benchmark data external rather than embedded.

Related errors

• zed benchmarks require build_id
• unknown benchmark '{benchmark_id}' (valid: {valid})
• unknown benchmark '{selector}' (valid: {valid})
• build ids may only contain lowercase letters, numbers, '.',
• unknown judge preset '{name}' (valid: {valid})

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/9dfdaf8ea1340cc6. Report an issue: GitHub.