zed-industries/zed · error · FileNotFoundError

benchmark dataset directory not found: {data_dir}

Error message

benchmark dataset directory not found: {data_dir}

What it means

Raised as FileNotFoundError by modal_app.provision_benchmark_dataset() after a successful git fetch/checkout of the dataset repo, when the expected task directory /tmp/datasets/{benchmark_id}/{data_dir} does not exist in the checked-out tree. The clone itself worked; the configured data_dir just does not match the repo layout at repo_ref. This is remote-side (inside the Modal controller), so it fails a run that already started.

Source

Thrown at crates/eval_cli/zed_eval/modal_app.py:127

    repo_ref = dataset.get("repo_ref") or "main"
    if not repo_url:
        raise ValueError(f"benchmark {benchmark['id']} path dataset requires repo_url")

    clone_dir = pathlib.Path(harness_command.dataset_clone_dir(benchmark))
    if clone_dir.exists():
        shutil.rmtree(clone_dir)
    clone_dir.mkdir(parents=True, exist_ok=True)
    log(f"Fetching {benchmark['id']} dataset {repo_url}@{repo_ref}")
    subprocess.run(["git", "init", "-q", str(clone_dir)], check=True)
    subprocess.run(
        ["git", "fetch", "--depth", "1", repo_url, repo_ref],
        cwd=clone_dir,
        check=True,
    )
    subprocess.run(["git", "checkout", "-q", "FETCH_HEAD"], cwd=clone_dir, check=True)
    data_dir = pathlib.Path(harness_command.dataset_path(benchmark))
    if not data_dir.exists():
        raise FileNotFoundError(f"benchmark dataset directory not found: {data_dir}")


def output_of(command: list[str]) -> str:
    try:
        return subprocess.run(
            command,
            check=True,
            capture_output=True,
            text=True,
        ).stdout.strip()
    except subprocess.CalledProcessError as error:
        return f"unknown ({error})"


@app.function(
    image=build_image,
    # Right-sized from the original 16 cpu / 32 GB after observing peak usage of
    # ~10 cores and ~10 GB. ephemeral_disk stays at Modal's 512 GiB floor (the

View on GitHub (pinned to bc538def45)

Solutions

1. Check the upstream repo at the pinned ref: git ls-remote / browse for the actual task directory, e.g. git ls-tree origin/main --name-only in a clone of the dataset repo.
2. Update data_dir (and/or repo_ref) in benchmarks.py to the current layout and redeploy the Modal app.
3. Pin repo_ref to a specific commit known to contain the directory so future upstream restructures cannot break provisioning.

Example fix

# before
DatasetRef(kind=DATASET_PATH, repo_url=SWE_ATLAS_REPO_URL,
           repo_ref="main", data_dir="data/tw")
# upstream moved tw tasks to data/test-writing -> FileNotFoundError

# after
DatasetRef(kind=DATASET_PATH, repo_url=SWE_ATLAS_REPO_URL,
           repo_ref="<pinned-commit-sha>", data_dir="data/test-writing")

# Pre-flight against the dataset repo before launching (local, cheap)
import subprocess

ref = dataset.get("repo_ref") or "main"
listing = subprocess.run(
    ["git", "ls-remote", dataset["repo_url"], ref], capture_output=True, text=True
)
if listing.returncode != 0:
    raise SystemExit(f"cannot reach dataset repo {dataset['repo_url']} at {ref}")

try:
    provision_benchmark_dataset(run_request, log)
except FileNotFoundError as error:
    log(f"dataset directory missing after clone: {error}")
    log("check data_dir/repo_ref against the upstream repo layout and redeploy")
    raise

When it happens

Trigger: Upstream dataset repo moved or renamed the task directory (SWE-Atlas moving data/tw, DeepSWE moving tasks/); data_dir typo in the registry; repo_ref pointing at a branch where the directory was removed or not yet created; a shallow fetch (--depth 1) succeeding at a ref that lacks the path.

Common situations: Tracking a moving branch ("main") for datasets — any upstream restructure breaks runs; pinning repo_ref to a stale tag after the repo reorganized; typos like "data//tw" or leading slashes producing a wrong joined path.

Related errors

• benchmark {benchmark['id']} path dataset requires repo_url
• registry dataset requires a name
• path dataset requires data_dir
• unsupported dataset kind: {kind}
• build {build_id} was not ready before the wait timeout

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/02fb0d7c77fda76b. Report an issue: GitHub.