{"record":{"id":"7657542f0c30f7c5","repo":"zed-industries/zed","slug":"path-dataset-requires-data-dir","errorCode":null,"errorMessage":"path dataset requires data_dir","messagePattern":"path dataset requires data_dir","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":34,"sourceCode":"def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:\n    block = run_request.get(\"benchmark\")\n    if not isinstance(block, dict):\n        raise ValueError(\"run request is missing a 'benchmark' block\")\n    return block\n\n\ndef dataset_args(benchmark: dict[str, Any]) -> list[str]:\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind == benchmarks.DATASET_REGISTRY:\n        name = dataset.get(\"name\")\n        if not name:\n            raise ValueError(\"registry dataset requires a name\")\n        return [\"-d\", name]\n    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        data_dir = dataset.get(\"data_dir\")\n        if not data_dir:\n            raise ValueError(\"path dataset requires data_dir\")\n        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n\n\ndef dataset_clone_dir(benchmark: dict[str, Any]) -> str:\n    \"\"\"Where the controller clones the dataset repo for path datasets.\"\"\"\n    return f\"/tmp/datasets/{benchmark['id']}\"\n\n\ndef dataset_path(benchmark: dict[str, Any]) -> str:\n    \"\"\"Repo-relative task directory inside the cloned dataset repo.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    return f\"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}\"\n\n\ndef harness_binary(benchmark: dict[str, Any]) -> str:\n    harness = benchmark.get(\"harness\")\n    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):","sourceCodeStart":16,"sourceCodeEnd":52,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/harness_command.py#L16-L52","documentation":"Raised by harness_command.dataset_args() when dataset.kind is \"path\" or \"pier_path\" but dataset.data_dir is missing or empty. Path datasets are cloned by the controller into /tmp/datasets/{benchmark_id} and passed to the harness as -p /tmp/datasets/{id}/{data_dir}, so the repo-relative task directory is mandatory. In-tree path benchmarks (swe-atlas-tw, deepswe) always set it.","triggerScenarios":"Hand-writing a path dataset block with repo_url but no data_dir; renaming the field (e.g. \"dir\" or \"task_dir\") when constructing the block; registering a new path benchmark whose DatasetRef omits data_dir.","commonSituations":"Adding a new git-backed benchmark and forgetting the sub-directory; upstream repo keeps tasks at the root so the author assumes data_dir is optional; copy-paste from a registry dataset entry where data_dir is legitimately None.","solutions":["Set dataset.data_dir to the repo-relative directory containing the tasks (e.g. \"data/tw\" for SWE-Atlas tw, \"tasks\" for DeepSWE).","Use benchmarks.DatasetRef(kind=..., repo_url=..., repo_ref=..., data_dir=...) plus benchmark_metadata() so fields cannot be dropped.","If the tasks actually live on the harness hub, switch kind to \"registry\" and provide name instead."],"exampleFix":"# before\n\"dataset\": {\"kind\": \"path\", \"repo_url\": \"https://github.com/scaleapi/SWE-Atlas.git\"}\n# -> ValueError: path dataset requires data_dir\n\n# after\n\"dataset\": {\n    \"kind\": \"path\",\n    \"repo_url\": \"https://github.com/scaleapi/SWE-Atlas.git\",\n    \"repo_ref\": \"main\",\n    \"data_dir\": \"data/tw\",\n}","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\ndataset = (run_request.get(\"benchmark\") or {}).get(\"dataset\") or {}\nif dataset.get(\"kind\") in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n    if not dataset.get(\"data_dir\"):\n        raise SystemExit(\"path dataset needs a repo-relative 'data_dir'\")","typeGuard":"def is_valid_path_dataset(dataset: dict) -> bool:\n    return dataset.get(\"kind\") in (\"path\", \"pier_path\") and bool(dataset.get(\"data_dir\"))","tryCatchPattern":"try:\n    args_ = harness_command.dataset_args(benchmark)\nexcept ValueError as error:\n    raise SystemExit(f\"benchmark dataset block invalid: {error}\")","preventionTips":["data_dir is repo-relative (e.g. data/tw, tasks); no leading slash, no absolute paths.","Always construct DatasetRef with explicit repo_url/repo_ref/data_dir for git-backed benchmarks.","Add a registry sanity test: every path-kind benchmark has repo_url and data_dir."],"tags":["validation","dataset","config"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}