{"record":{"id":"02fb0d7c77fda76b","repo":"zed-industries/zed","slug":"benchmark-dataset-directory-not-found-data-dir","errorCode":null,"errorMessage":"benchmark dataset directory not found: {data_dir}","messagePattern":"benchmark dataset directory not found: (.+?)","errorType":"exception","errorClass":"FileNotFoundError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/modal_app.py","lineNumber":127,"sourceCode":"    repo_ref = dataset.get(\"repo_ref\") or \"main\"\n    if not repo_url:\n        raise ValueError(f\"benchmark {benchmark['id']} path dataset requires repo_url\")\n\n    clone_dir = pathlib.Path(harness_command.dataset_clone_dir(benchmark))\n    if clone_dir.exists():\n        shutil.rmtree(clone_dir)\n    clone_dir.mkdir(parents=True, exist_ok=True)\n    log(f\"Fetching {benchmark['id']} dataset {repo_url}@{repo_ref}\")\n    subprocess.run([\"git\", \"init\", \"-q\", str(clone_dir)], check=True)\n    subprocess.run(\n        [\"git\", \"fetch\", \"--depth\", \"1\", repo_url, repo_ref],\n        cwd=clone_dir,\n        check=True,\n    )\n    subprocess.run([\"git\", \"checkout\", \"-q\", \"FETCH_HEAD\"], cwd=clone_dir, check=True)\n    data_dir = pathlib.Path(harness_command.dataset_path(benchmark))\n    if not data_dir.exists():\n        raise FileNotFoundError(f\"benchmark dataset directory not found: {data_dir}\")\n\n\ndef output_of(command: list[str]) -> str:\n    try:\n        return subprocess.run(\n            command,\n            check=True,\n            capture_output=True,\n            text=True,\n        ).stdout.strip()\n    except subprocess.CalledProcessError as error:\n        return f\"unknown ({error})\"\n\n\n@app.function(\n    image=build_image,\n    # Right-sized from the original 16 cpu / 32 GB after observing peak usage of\n    # ~10 cores and ~10 GB. ephemeral_disk stays at Modal's 512 GiB floor (the","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/modal_app.py#L109-L145","documentation":"Raised as FileNotFoundError by modal_app.provision_benchmark_dataset() after a successful git fetch/checkout of the dataset repo, when the expected task directory /tmp/datasets/{benchmark_id}/{data_dir} does not exist in the checked-out tree. The clone itself worked; the configured data_dir just does not match the repo layout at repo_ref. This is remote-side (inside the Modal controller), so it fails a run that already started.","triggerScenarios":"Upstream dataset repo moved or renamed the task directory (SWE-Atlas moving data/tw, DeepSWE moving tasks/); data_dir typo in the registry; repo_ref pointing at a branch where the directory was removed or not yet created; a shallow fetch (--depth 1) succeeding at a ref that lacks the path.","commonSituations":"Tracking a moving branch (\"main\") for datasets — any upstream restructure breaks runs; pinning repo_ref to a stale tag after the repo reorganized; typos like \"data//tw\" or leading slashes producing a wrong joined path.","solutions":["Check the upstream repo at the pinned ref: git ls-remote / browse for the actual task directory, e.g. git ls-tree origin/main --name-only in a clone of the dataset repo.","Update data_dir (and/or repo_ref) in benchmarks.py to the current layout and redeploy the Modal app.","Pin repo_ref to a specific commit known to contain the directory so future upstream restructures cannot break provisioning."],"exampleFix":"# before\nDatasetRef(kind=DATASET_PATH, repo_url=SWE_ATLAS_REPO_URL,\n           repo_ref=\"main\", data_dir=\"data/tw\")\n# upstream moved tw tasks to data/test-writing -> FileNotFoundError\n\n# after\nDatasetRef(kind=DATASET_PATH, repo_url=SWE_ATLAS_REPO_URL,\n           repo_ref=\"<pinned-commit-sha>\", data_dir=\"data/test-writing\")","handlingStrategy":"try-catch","validationCode":"# Pre-flight against the dataset repo before launching (local, cheap)\nimport subprocess\n\nref = dataset.get(\"repo_ref\") or \"main\"\nlisting = subprocess.run(\n    [\"git\", \"ls-remote\", dataset[\"repo_url\"], ref], capture_output=True, text=True\n)\nif listing.returncode != 0:\n    raise SystemExit(f\"cannot reach dataset repo {dataset['repo_url']} at {ref}\")","typeGuard":null,"tryCatchPattern":"try:\n    provision_benchmark_dataset(run_request, log)\nexcept FileNotFoundError as error:\n    log(f\"dataset directory missing after clone: {error}\")\n    log(\"check data_dir/repo_ref against the upstream repo layout and redeploy\")\n    raise","preventionTips":["Pin dataset repo_ref to a commit instead of a moving branch so upstream restructures cannot break provisioning.","Verify the task directory exists at the ref (git ls-tree in a clone) whenever you touch dataset metadata.","Treat any change to an upstream dataset repo layout as a breaking event: update data_dir and redeploy."],"tags":["dataset","git","upstream-drift","controller"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}