zed-industries/zed · error · ValueError

registry dataset requires a name

Error message

registry dataset requires a name

What it means

Raised by harness_command.dataset_args() when the benchmark's dataset.kind == "registry" but dataset.name is missing or empty. Registry datasets are pulled from the harness hub via the -d <name> flag, so a hub dataset name is mandatory. All in-tree registry benchmarks (swe-atlas-qna, swe-atlas-rf, terminal-bench-2.1) carry a name; this fires only for hand-authored or mutated benchmark blocks.

Source

Thrown at crates/eval_cli/zed_eval/harness_command.py:29

from typing import Any

from . import benchmarks, config


def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:
    block = run_request.get("benchmark")
    if not isinstance(block, dict):
        raise ValueError("run request is missing a 'benchmark' block")
    return block


def dataset_args(benchmark: dict[str, Any]) -> list[str]:
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind == benchmarks.DATASET_REGISTRY:
        name = dataset.get("name")
        if not name:
            raise ValueError("registry dataset requires a name")
        return ["-d", name]
    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        data_dir = dataset.get("data_dir")
        if not data_dir:
            raise ValueError("path dataset requires data_dir")
        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")


def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
    """Where the controller clones the dataset repo for path datasets."""
    return f"/tmp/datasets/{benchmark['id']}"


def dataset_path(benchmark: dict[str, Any]) -> str:
    """Repo-relative task directory inside the cloned dataset repo."""
    dataset = benchmark.get("dataset") or {}
    return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"

View on GitHub (pinned to bc538def45)

Solutions

1. Set dataset.name to the harness hub dataset name, e.g. "scale-ai/swe-atlas-qna" or "terminal-bench/terminal-bench-2-1".
2. Prefer generating the block with benchmarks.benchmark_metadata() so the field layout is always correct.
3. If your tasks live in a git repo instead of the hub, use kind="path"/"pier_path" with repo_url and data_dir rather than registry.

Example fix

# before
"dataset": {"kind": "registry"}  # -> ValueError: registry dataset requires a name

# after
"dataset": {"kind": "registry", "name": "terminal-bench/terminal-bench-2-1"}

from zed_eval import benchmarks

dataset = (run_request.get("benchmark") or {}).get("dataset") or {}
if dataset.get("kind") == benchmarks.DATASET_REGISTRY and not dataset.get("name"):
    raise SystemExit("registry dataset needs a hub 'name' (e.g. scale-ai/swe-atlas-qna)")

def is_valid_registry_dataset(dataset: dict) -> bool:
    return dataset.get("kind") == "registry" and bool(dataset.get("name"))

try:
    args_ = harness_command.dataset_args(benchmark)
except ValueError as error:
    raise SystemExit(f"benchmark dataset block invalid: {error}")

When it happens

Trigger: Building a benchmark block with kind="registry" but no name key; copying benchmark metadata and deleting/rename-casing the name field; programmatic dataset blocks that set kind but rely on a separate variable for the hub name that is None.

Common situations: Adding a new benchmark to the registry and forgetting DatasetRef(name=...); typos like "dataset_name" instead of "name" when hand-writing the block; switching a benchmark from path to registry kind without adding the hub name.

Related errors

• path dataset requires data_dir
• unknown judge preset '{name}' (valid: {valid})
• unsupported dataset kind: {kind}
• unsupported harness: {harness}
• benchmark {benchmark['id']} path dataset requires repo_url

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/d695f9fbc9420c81. Report an issue: GitHub.