{"record":{"id":"9dfdaf8ea1340cc6","repo":"zed-industries/zed","slug":"run-request-is-missing-a-benchmark-block","errorCode":null,"errorMessage":"run request is missing a 'benchmark' block","messagePattern":"run request is missing a 'benchmark' block","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":19,"sourceCode":"\"\"\"Build the harness command (Harbor or Pier) for a benchmark run request.\n\nThe command is driven by the self-describing `benchmark` block embedded in a run\nrequest (see `benchmarks.benchmark_metadata`), so it works for any registered\nbenchmark without a separate experiment registry lookup.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport json\nfrom typing import Any\n\nfrom . import benchmarks, config\n\n\ndef _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:\n    block = run_request.get(\"benchmark\")\n    if not isinstance(block, dict):\n        raise ValueError(\"run request is missing a 'benchmark' block\")\n    return block\n\n\ndef dataset_args(benchmark: dict[str, Any]) -> list[str]:\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind == benchmarks.DATASET_REGISTRY:\n        name = dataset.get(\"name\")\n        if not name:\n            raise ValueError(\"registry dataset requires a name\")\n        return [\"-d\", name]\n    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        data_dir = dataset.get(\"data_dir\")\n        if not data_dir:\n            raise ValueError(\"path dataset requires data_dir\")\n        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n","sourceCodeStart":1,"sourceCodeEnd":37,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/harness_command.py#L1-L37","documentation":"Raised by harness_command._benchmark_block() when run_request.get(\"benchmark\") is missing, None, or not a dict. Every run request produced by launch.build_benchmark_run_request embeds a self-describing benchmark block from benchmarks.benchmark_metadata(); both build_harness_command() and run_metadata() (called by the controller's write_run_inputs) require it so the harness command can be built without a separate registry lookup.","triggerScenarios":"Calling harness_command.build_harness_command(run_request, jobs_dir) or harness_command.run_metadata(run_request) with a hand-constructed dict that lacks the \"benchmark\" key; loading a saved request.json from the volume and mutating/dropping fields before replaying it; a rejudge-style request (which carries no benchmark block) accidentally fed to the harness-command builder.","commonSituations":"Scripting the controller directly instead of going through zed-eval run; partial-JSON round-trips where the block was serialized under a different key; code written against an older request schema that kept benchmark data external rather than embedded.","solutions":["Embed the registry block before dispatching: run_request[\"benchmark\"] = benchmarks.benchmark_metadata(benchmarks.get_benchmark(benchmark_id)).","If replaying a stored request, use the untouched /data/runs/.../request.json rather than a hand-edited copy.","Add an assertion before spawning the controller: assert isinstance(run_request.get(\"benchmark\"), dict)."],"exampleFix":"# before\nrun_request = {\n    \"run_id\": \"r1\",\n    \"namespace\": \"me\",\n    # no \"benchmark\" block -> ValueError in build_harness_command\n}\n\n# after\nfrom zed_eval import benchmarks\nrun_request[\"benchmark\"] = benchmarks.benchmark_metadata(\n    benchmarks.get_benchmark(\"swe-atlas-rf\")\n)","handlingStrategy":"type-guard","validationCode":"from zed_eval import benchmarks\n\nif not isinstance(run_request.get(\"benchmark\"), dict):\n    run_request[\"benchmark\"] = benchmarks.benchmark_metadata(\n        benchmarks.get_benchmark(benchmark_id)\n    )","typeGuard":"from typing import Any\n\ndef has_benchmark_block(run_request: dict[str, Any]) -> bool:\n    \"\"\"True when the run request carries a usable self-describing benchmark block.\"\"\"\n    return isinstance(run_request.get(\"benchmark\"), dict)","tryCatchPattern":"try:\n    command = harness_command.build_harness_command(run_request, jobs_dir)\nexcept ValueError as error:\n    raise SystemExit(f\"run request rejected: {error}\")","preventionTips":["Always build run requests via build_benchmark_run_request; it embeds the benchmark block.","When replaying stored requests, use the untouched request.json from the volume.","Assert isinstance(run_request.get(\"benchmark\"), dict) before spawning any controller."],"tags":["validation","api-contract","run-request"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}