zed-industries/zed · error · ValueError

benchmark {benchmark['id']} path dataset requires repo_url

Error message

benchmark {benchmark['id']} path dataset requires repo_url

What it means

Raised by modal_app.provision_benchmark_dataset(), running inside the Modal controller, when a benchmark's dataset.kind is "path" or "pier_path" but dataset.repo_url is missing. Path datasets are provisioned by git init + git fetch --depth 1 of repo_url at repo_ref (default "main") into /tmp/datasets/{id}, so a clone URL is mandatory. Registry datasets return early and never hit this. All in-tree path benchmarks (swe-atlas-tw, deepswe) define repo_url in benchmarks.py.

Source

Thrown at crates/eval_cli/zed_eval/modal_app.py:111

        "echo 'WARNING: pier install failed; DeepSWE runs will not work'",
    )
)


def provision_benchmark_dataset(run_request: dict[str, Any], log: Any) -> None:
    """Clone a path-backed benchmark dataset (SWE-Atlas tw, DeepSWE) into the
    location the harness command expects. Registry datasets need no provisioning;
    the harness pulls them from its hub."""
    benchmark = run_request["benchmark"]
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind not in ("path", "pier_path"):
        return

    repo_url = dataset.get("repo_url")
    repo_ref = dataset.get("repo_ref") or "main"
    if not repo_url:
        raise ValueError(f"benchmark {benchmark['id']} path dataset requires repo_url")

    clone_dir = pathlib.Path(harness_command.dataset_clone_dir(benchmark))
    if clone_dir.exists():
        shutil.rmtree(clone_dir)
    clone_dir.mkdir(parents=True, exist_ok=True)
    log(f"Fetching {benchmark['id']} dataset {repo_url}@{repo_ref}")
    subprocess.run(["git", "init", "-q", str(clone_dir)], check=True)
    subprocess.run(
        ["git", "fetch", "--depth", "1", repo_url, repo_ref],
        cwd=clone_dir,
        check=True,
    )
    subprocess.run(["git", "checkout", "-q", "FETCH_HEAD"], cwd=clone_dir, check=True)
    data_dir = pathlib.Path(harness_command.dataset_path(benchmark))
    if not data_dir.exists():
        raise FileNotFoundError(f"benchmark dataset directory not found: {data_dir}")

View on GitHub (pinned to bc538def45)

Solutions

1. Set dataset.repo_url to a fetchable https git URL (e.g. https://github.com/datacurve-ai/deep-swe.git) in the benchmark's DatasetRef.
2. Redeploy the Modal app after editing benchmarks.py so the controller sees the fixed metadata, then relaunch.
3. If the tasks are on the harness hub, use kind="registry" with name and no provisioning is needed.

Example fix

# before
DatasetRef(kind=DATASET_PATH, data_dir="data/tw")  # no repo_url
# controller: ValueError: benchmark swe-atlas-tw path dataset requires repo_url

# after
DatasetRef(
    kind=DATASET_PATH,
    repo_url=SWE_ATLAS_REPO_URL,
    repo_ref=SWE_ATLAS_REPO_REF,
    data_dir="data/tw",
)

from zed_eval import benchmarks

for benchmark in benchmarks.BENCHMARKS.values():
    dataset = benchmark.dataset
    if dataset.kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        assert dataset.repo_url, f"{benchmark.id} path dataset missing repo_url"
        assert dataset.data_dir, f"{benchmark.id} path dataset missing data_dir"

def is_provisionable_dataset(dataset: dict) -> bool:
    kind = dataset.get("kind")
    if kind in ("path", "pier_path"):
        return bool(dataset.get("repo_url"))
    return kind == "registry"

When it happens

Trigger: Registering a custom path/pier_path benchmark whose DatasetRef omits repo_url; hand-editing benchmark metadata and dropping repo_url; a run request whose embedded benchmark block was built against a modified registry. The failure happens remotely after the controller has already started and (if applicable) waited for the build.

Common situations: Adding a new git-backed benchmark and forgetting the URL; assuming repo_url is optional because repo_ref has a default; forking the registry and renaming fields without redeploying.

Related errors

• build ids may only contain lowercase letters, numbers, '.',
• registry dataset requires a name
• path dataset requires data_dir
• unsupported dataset kind: {kind}
• benchmark dataset directory not found: {data_dir}

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/b025e44facb14f60. Report an issue: GitHub.