zed-industries/zed · error · ValueError

path dataset requires data_dir

Error message

path dataset requires data_dir

What it means

Raised by dataset_args() in zed_eval/harness_command.py when a benchmark's dataset block declares kind 'path' or 'pier_path' but carries no data_dir. data_dir is the repo-relative task directory that gets appended to /tmp/datasets/<benchmark_id> to form the -p argument passed to Harbor/Pier, so without it the harness command cannot be built. The value originates from the DatasetRef dataclass in benchmarks.py and is embedded verbatim into the run request's benchmark block via benchmark_metadata().

Solutions

  1. If you control the benchmark registration, add the data_dir field to its DatasetRef in crates/eval_cli/zed_eval/benchmarks.py (e.g. data_dir="data/tw" like swe-atlas-tw, or "tasks" like deepswe)
  2. If the benchmark pulls tasks from the harness hub instead of a git repo, change the DatasetRef to kind=DATASET_REGISTRY with a name so it emits -d <name>
  3. If the block arrives from a run request you built programmatically, set run_request['benchmark']['dataset']['data_dir'] before invoking build_harness_command or spawning run_controller
  4. If you genuinely have no sub-directory (tasks at repo root), set data_dir to the root dir name that exists in the cloned repo (e.g. deepswe uses "tasks")

Example fix

# before (benchmarks.py)
"my-bench": Benchmark(
    id="my-bench",
    harness=HARNESS_HARBOR,
    dataset=DatasetRef(kind=DATASET_PATH, repo_url=MY_REPO_URL),
    ...
)
# after
"my-bench": Benchmark(
    id="my-bench",
    harness=HARNESS_HARBOR,
    dataset=DatasetRef(
        kind=DATASET_PATH,
        repo_url=MY_REPO_URL,
        repo_ref="main",
        data_dir="tasks",
    ),
    ...
)
Defensive patterns

Strategy: validation

Validate before calling

from zed_eval import benchmarks

def dataset_args_valid(benchmark: dict) -> bool:
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        return bool(dataset.get("data_dir"))
    if kind == benchmarks.DATASET_REGISTRY:
        return bool(dataset.get("name"))
    return False

# before building the harness command:
assert dataset_args_valid(run_request["benchmark"]), "dataset block incomplete"

Type guard

from zed_eval import benchmarks

def has_complete_path_dataset(
    benchmark: dict,
) -> tuple[bool, str | None]:
    """Returns (ok, resolved_data_dir) for path-style datasets."""
    dataset = benchmark.get("dataset") or {}
    if dataset.get("kind") not in (
        benchmarks.DATASET_PATH,
        benchmarks.DATASET_PIER_PATH,
    ):
        return False, None
    data_dir = dataset.get("data_dir")
    return (isinstance(data_dir, str) and data_dir != "", data_dir)

Try / catch

try:
    args = harness_command.dataset_args(benchmark)
except ValueError as error:
    raise SystemExit(f"benchmark config invalid: {error}") from error

Prevention

When it happens

Trigger: Calling harness_command.build_harness_command(run_request, jobs_dir) (or run_controller on Modal) where run_request['benchmark']['dataset']['kind'] is DATASET_PATH ('path') or DATASET_PIER_PATH ('pier_path') and dataset['data_dir'] is None or ''. Concretely: registering a Benchmark with DatasetRef(kind=DATASET_PATH, repo_url=...) while forgetting data_dir, or hand-building a run request dict whose benchmark block omits data_dir.

Common situations: Adding a new path-backed benchmark to BENCHMARKS in benchmarks.py and omitting data_dir; copying the swe-atlas-tw or deepswe entry and deleting the data_dir line; mutating a benchmark block from JSON/CLI input that drops the field; refactoring DatasetRef defaults so data_dir silently defaults to None.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of zed-industries/zed@f4178619ac (2026-08-20). Data as JSON: /api/errors/7657542f0c30f7c5. Report an issue: GitHub.

Appendix: source

Thrown at crates/eval_cli/zed_eval/harness_command.py:34

def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:
    block = run_request.get("benchmark")
    if not isinstance(block, dict):
        raise ValueError("run request is missing a 'benchmark' block")
    return block


def dataset_args(benchmark: dict[str, Any]) -> list[str]:
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind == benchmarks.DATASET_REGISTRY:
        name = dataset.get("name")
        if not name:
            raise ValueError("registry dataset requires a name")
        return ["-d", name]
    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        data_dir = dataset.get("data_dir")
        if not data_dir:
            raise ValueError("path dataset requires data_dir")
        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")


def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
    """Where the controller clones the dataset repo for path datasets."""
    return f"/tmp/datasets/{benchmark['id']}"


def dataset_path(benchmark: dict[str, Any]) -> str:
    """Repo-relative task directory inside the cloned dataset repo."""
    dataset = benchmark.get("dataset") or {}
    return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"


def harness_binary(benchmark: dict[str, Any]) -> str:
    harness = benchmark.get("harness")
    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):

View on GitHub (pinned to f4178619ac)