zed-industries/zed · error · ValueError

path dataset requires data_dir

Error message

path dataset requires data_dir

What it means

Raised by harness_command.dataset_args() when dataset.kind is "path" or "pier_path" but dataset.data_dir is missing or empty. Path datasets are cloned by the controller into /tmp/datasets/{benchmark_id} and passed to the harness as -p /tmp/datasets/{id}/{data_dir}, so the repo-relative task directory is mandatory. In-tree path benchmarks (swe-atlas-tw, deepswe) always set it.

Source

Thrown at crates/eval_cli/zed_eval/harness_command.py:34

def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:
    block = run_request.get("benchmark")
    if not isinstance(block, dict):
        raise ValueError("run request is missing a 'benchmark' block")
    return block


def dataset_args(benchmark: dict[str, Any]) -> list[str]:
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind == benchmarks.DATASET_REGISTRY:
        name = dataset.get("name")
        if not name:
            raise ValueError("registry dataset requires a name")
        return ["-d", name]
    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        data_dir = dataset.get("data_dir")
        if not data_dir:
            raise ValueError("path dataset requires data_dir")
        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")


def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
    """Where the controller clones the dataset repo for path datasets."""
    return f"/tmp/datasets/{benchmark['id']}"


def dataset_path(benchmark: dict[str, Any]) -> str:
    """Repo-relative task directory inside the cloned dataset repo."""
    dataset = benchmark.get("dataset") or {}
    return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"


def harness_binary(benchmark: dict[str, Any]) -> str:
    harness = benchmark.get("harness")
    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):

View on GitHub (pinned to bc538def45)

Solutions

1. Set dataset.data_dir to the repo-relative directory containing the tasks (e.g. "data/tw" for SWE-Atlas tw, "tasks" for DeepSWE).
2. Use benchmarks.DatasetRef(kind=..., repo_url=..., repo_ref=..., data_dir=...) plus benchmark_metadata() so fields cannot be dropped.
3. If the tasks actually live on the harness hub, switch kind to "registry" and provide name instead.

Example fix

# before
"dataset": {"kind": "path", "repo_url": "https://github.com/scaleapi/SWE-Atlas.git"}
# -> ValueError: path dataset requires data_dir

# after
"dataset": {
    "kind": "path",
    "repo_url": "https://github.com/scaleapi/SWE-Atlas.git",
    "repo_ref": "main",
    "data_dir": "data/tw",
}

from zed_eval import benchmarks

dataset = (run_request.get("benchmark") or {}).get("dataset") or {}
if dataset.get("kind") in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
    if not dataset.get("data_dir"):
        raise SystemExit("path dataset needs a repo-relative 'data_dir'")

def is_valid_path_dataset(dataset: dict) -> bool:
    return dataset.get("kind") in ("path", "pier_path") and bool(dataset.get("data_dir"))

try:
    args_ = harness_command.dataset_args(benchmark)
except ValueError as error:
    raise SystemExit(f"benchmark dataset block invalid: {error}")

When it happens

Trigger: Hand-writing a path dataset block with repo_url but no data_dir; renaming the field (e.g. "dir" or "task_dir") when constructing the block; registering a new path benchmark whose DatasetRef omits data_dir.

Common situations: Adding a new git-backed benchmark and forgetting the sub-directory; upstream repo keeps tasks at the root so the author assumes data_dir is optional; copy-paste from a registry dataset entry where data_dir is legitimately None.

Related errors

• registry dataset requires a name
• unknown judge preset '{name}' (valid: {valid})
• unsupported dataset kind: {kind}
• unsupported harness: {harness}
• benchmark {benchmark['id']} path dataset requires repo_url

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/7657542f0c30f7c5. Report an issue: GitHub.