zed-industries/zed · error · ValueError

unsupported dataset kind: {kind}

Error message

unsupported dataset kind: {kind}

What it means

Raised by harness_command.dataset_args() when dataset.kind is neither "registry", "path", nor "pier_path" — including when the dataset block is absent entirely (kind is None). These three kinds are the only provisioning strategies the controller knows: hub pull (-d), git clone for Harbor (-p), git clone for Pier (-p under pier). The kind constants live in benchmarks.py:25-27.

Source

Thrown at crates/eval_cli/zed_eval/harness_command.py:36

    if not isinstance(block, dict):
        raise ValueError("run request is missing a 'benchmark' block")
    return block


def dataset_args(benchmark: dict[str, Any]) -> list[str]:
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind == benchmarks.DATASET_REGISTRY:
        name = dataset.get("name")
        if not name:
            raise ValueError("registry dataset requires a name")
        return ["-d", name]
    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        data_dir = dataset.get("data_dir")
        if not data_dir:
            raise ValueError("path dataset requires data_dir")
        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")


def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
    """Where the controller clones the dataset repo for path datasets."""
    return f"/tmp/datasets/{benchmark['id']}"


def dataset_path(benchmark: dict[str, Any]) -> str:
    """Repo-relative task directory inside the cloned dataset repo."""
    dataset = benchmark.get("dataset") or {}
    return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"


def harness_binary(benchmark: dict[str, Any]) -> str:
    harness = benchmark.get("harness")
    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):
        raise ValueError(f"unsupported harness: {harness}")
    return harness

View on GitHub (pinned to bc538def45)

Solutions

1. Set dataset.kind to exactly "registry", "path", or "pier_path".
2. Regenerate the benchmark block from the registry (benchmarks.benchmark_metadata) matching the zed_eval version you deploy.
3. Redeploy the Modal app (zed-eval deploy) so the controller's harness_command knows the same kinds as your client.

Example fix

# before
"dataset": {"kind": "hub", "name": "scale-ai/swe-atlas-qna"}
# -> ValueError: unsupported dataset kind: hub

# after
"dataset": {"kind": "registry", "name": "scale-ai/swe-atlas-qna"}

from zed_eval import benchmarks

VALID_KINDS = {
    benchmarks.DATASET_REGISTRY,
    benchmarks.DATASET_PATH,
    benchmarks.DATASET_PIER_PATH,
}
if dataset.get("kind") not in VALID_KINDS:
    raise SystemExit(
        f"dataset.kind must be one of {sorted(VALID_KINDS)}, got {dataset.get('kind')!r}"
    )

def is_known_dataset_kind(kind: object) -> bool:
    return kind in ("registry", "path", "pier_path")

try:
    args_ = harness_command.dataset_args(benchmark)
except ValueError as error:
    raise SystemExit(str(error))  # echo kind so the mismatch is obvious

When it happens

Trigger: Spelling mistakes in kind ("Registry", "git", "local", "pier-path"); a run request produced by a newer CLI that added a new dataset kind, replayed on an older deployed controller; a benchmark block whose dataset key was dropped so dataset defaults to {} and kind is None.

Common situations: Version skew between the local CLI and the deployed Modal app after new dataset kinds are introduced; hand-rolled benchmark metadata; renaming kind values in a fork without redeploying the controller image.

Related errors

• registry dataset requires a name
• path dataset requires data_dir
• benchmark {benchmark['id']} path dataset requires repo_url
• unknown benchmark '{benchmark_id}' (valid: {valid})
• unknown benchmark '{selector}' (valid: {valid})

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/dfc4d1c83a71de21. Report an issue: GitHub.