{"record":{"id":"7657542f0c30f7c5","repo":"zed-industries/zed","slug":"path-dataset-requires-data-dir","errorCode":null,"errorMessage":"path dataset requires data_dir","messagePattern":"path dataset requires data_dir","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":34,"sourceCode":"def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:\n    block = run_request.get(\"benchmark\")\n    if not isinstance(block, dict):\n        raise ValueError(\"run request is missing a 'benchmark' block\")\n    return block\n\n\ndef dataset_args(benchmark: dict[str, Any]) -> list[str]:\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind == benchmarks.DATASET_REGISTRY:\n        name = dataset.get(\"name\")\n        if not name:\n            raise ValueError(\"registry dataset requires a name\")\n        return [\"-d\", name]\n    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        data_dir = dataset.get(\"data_dir\")\n        if not data_dir:\n            raise ValueError(\"path dataset requires data_dir\")\n        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n\n\ndef dataset_clone_dir(benchmark: dict[str, Any]) -> str:\n    \"\"\"Where the controller clones the dataset repo for path datasets.\"\"\"\n    return f\"/tmp/datasets/{benchmark['id']}\"\n\n\ndef dataset_path(benchmark: dict[str, Any]) -> str:\n    \"\"\"Repo-relative task directory inside the cloned dataset repo.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    return f\"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}\"\n\n\ndef harness_binary(benchmark: dict[str, Any]) -> str:\n    harness = benchmark.get(\"harness\")\n    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):","sourceCodeStart":16,"sourceCodeEnd":52,"githubUrl":"https://github.com/zed-industries/zed/blob/f4178619acd0d47ea1f76a2025c42962c6d6638c/crates/eval_cli/zed_eval/harness_command.py#L16-L52","documentation":"Raised by dataset_args() in zed_eval/harness_command.py when a benchmark's dataset block declares kind 'path' or 'pier_path' but carries no data_dir. data_dir is the repo-relative task directory that gets appended to /tmp/datasets/<benchmark_id> to form the -p argument passed to Harbor/Pier, so without it the harness command cannot be built. The value originates from the DatasetRef dataclass in benchmarks.py and is embedded verbatim into the run request's benchmark block via benchmark_metadata().","triggerScenarios":"Calling harness_command.build_harness_command(run_request, jobs_dir) (or run_controller on Modal) where run_request['benchmark']['dataset']['kind'] is DATASET_PATH ('path') or DATASET_PIER_PATH ('pier_path') and dataset['data_dir'] is None or ''. Concretely: registering a Benchmark with DatasetRef(kind=DATASET_PATH, repo_url=...) while forgetting data_dir, or hand-building a run request dict whose benchmark block omits data_dir.","commonSituations":"Adding a new path-backed benchmark to BENCHMARKS in benchmarks.py and omitting data_dir; copying the swe-atlas-tw or deepswe entry and deleting the data_dir line; mutating a benchmark block from JSON/CLI input that drops the field; refactoring DatasetRef defaults so data_dir silently defaults to None.","solutions":["If you control the benchmark registration, add the data_dir field to its DatasetRef in crates/eval_cli/zed_eval/benchmarks.py (e.g. data_dir=\"data/tw\" like swe-atlas-tw, or \"tasks\" like deepswe)","If the benchmark pulls tasks from the harness hub instead of a git repo, change the DatasetRef to kind=DATASET_REGISTRY with a name so it emits -d <name>","If the block arrives from a run request you built programmatically, set run_request['benchmark']['dataset']['data_dir'] before invoking build_harness_command or spawning run_controller","If you genuinely have no sub-directory (tasks at repo root), set data_dir to the root dir name that exists in the cloned repo (e.g. deepswe uses \"tasks\")"],"exampleFix":"# before (benchmarks.py)\n\"my-bench\": Benchmark(\n    id=\"my-bench\",\n    harness=HARNESS_HARBOR,\n    dataset=DatasetRef(kind=DATASET_PATH, repo_url=MY_REPO_URL),\n    ...\n)\n# after\n\"my-bench\": Benchmark(\n    id=\"my-bench\",\n    harness=HARNESS_HARBOR,\n    dataset=DatasetRef(\n        kind=DATASET_PATH,\n        repo_url=MY_REPO_URL,\n        repo_ref=\"main\",\n        data_dir=\"tasks\",\n    ),\n    ...\n)","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\ndef dataset_args_valid(benchmark: dict) -> bool:\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        return bool(dataset.get(\"data_dir\"))\n    if kind == benchmarks.DATASET_REGISTRY:\n        return bool(dataset.get(\"name\"))\n    return False\n\n# before building the harness command:\nassert dataset_args_valid(run_request[\"benchmark\"]), \"dataset block incomplete\"","typeGuard":"from zed_eval import benchmarks\n\ndef has_complete_path_dataset(\n    benchmark: dict,\n) -> tuple[bool, str | None]:\n    \"\"\"Returns (ok, resolved_data_dir) for path-style datasets.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    if dataset.get(\"kind\") not in (\n        benchmarks.DATASET_PATH,\n        benchmarks.DATASET_PIER_PATH,\n    ):\n        return False, None\n    data_dir = dataset.get(\"data_dir\")\n    return (isinstance(data_dir, str) and data_dir != \"\", data_dir)","tryCatchPattern":"try:\n    args = harness_command.dataset_args(benchmark)\nexcept ValueError as error:\n    raise SystemExit(f\"benchmark config invalid: {error}\") from error","preventionTips":["Always construct benchmark blocks via benchmarks.benchmark_metadata(benchmark) instead of hand-writing dicts","Register new benchmarks next to the existing entries in benchmarks.py so DatasetRef fields are copy-paste complete","Add a unit test asserting every entry in BENCHMARKS has the fields its dataset kind requires"],"tags":["python","zed-eval","benchmark-config","validation","harness"],"backgroundTag":"missing-required-config-field","analyzedSha":"f4178619acd0d47ea1f76a2025c42962c6d6638c","analyzedAt":"2026-08-20T19:29:52.058Z","contentChangedAt":"2026-08-20T19:29:52.058Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}