zed-industries/zed · error · ValueError

unsupported dataset kind

Error message

unsupported dataset kind: {kind}

What it means

Raised by dataset_args() in zed_eval/harness_command.py when the benchmark's dataset kind is none of the three supported constants: 'registry', 'path', or 'pier_path' (benchmarks.DATASET_REGISTRY/DATASET_PATH/DATASET_PIER_PATH). The function maps kind to a CLI flag (-d <name> for registry, -p <dir> for path kinds); any other value has no mapping, so it refuses rather than guessing.

Solutions

  1. Set dataset kind to one of the exact strings 'registry', 'path', or 'pier_path' — preferably via the benchmarks.DATASET_* constants so typos cannot compile/lint past you
  2. If you introduced a new kind in benchmarks.py, add the corresponding branch to dataset_args() in harness_command.py returning the right -d/-p args
  3. If the block came from a stored run request JSON, fix the kind at the source and regenerate the request via benchmark_metadata()
  4. If kind is None, the whole dataset sub-block was dropped — restore it (benchmark.get('dataset') or {} masks the absence until this check fires)

Example fix

# before
benchmark["dataset"] = {"kind": "local", "data_dir": "data/tw"}
# after
from . import benchmarks
benchmark["dataset"] = {
    "kind": benchmarks.DATASET_PATH,
    "data_dir": "data/tw",
}
Defensive patterns

Strategy: validation

Validate before calling

from zed_eval import benchmarks

VALID_KINDS = {
    benchmarks.DATASET_REGISTRY,
    benchmarks.DATASET_PATH,
    benchmarks.DATASET_PIER_PATH,
}

kind = (run_request["benchmark"].get("dataset") or {}).get("kind")
if kind not in VALID_KINDS:
    raise SystemExit(
        f"dataset.kind must be one of {sorted(VALID_KINDS)}, got {kind!r}"
    )

Type guard

from zed_eval import benchmarks

def is_known_dataset_kind(benchmark: dict) -> bool:
    kind = (benchmark.get("dataset") or {}).get("kind")
    return kind in (
        benchmarks.DATASET_REGISTRY,
        benchmarks.DATASET_PATH,
        benchmarks.DATASET_PIER_PATH,
    )

Try / catch

try:
    harness_command.dataset_args(benchmark)
except ValueError as error:
    # error message names the unsupported kind — surface it to the operator
    log(f"rejecting benchmark {benchmark.get('id')}: {error}")
    return

Prevention

When it happens

Trigger: run_request['benchmark']['dataset']['kind'] is missing (None), misspelled ('Path', 'local', 'git'), or a genuinely new kind added to benchmarks.py without extending dataset_args(). Also triggered by hand-crafted or JSON-loaded benchmark blocks that bypass benchmarks.benchmark_metadata().

Common situations: Adding a new dataset provisioning mode (e.g. s3-backed) and forgetting the branch in dataset_args; typos when a benchmark block is edited as raw JSON; older run requests replayed against a newer zed_eval whose kind vocabulary changed; test fixtures inventing ad-hoc kinds.

Related errors

AI-assisted analysis of zed-industries/zed@f4178619ac (2026-08-20). Data as JSON: /api/errors/dfc4d1c83a71de21. Report an issue: GitHub.

Appendix: source

Thrown at crates/eval_cli/zed_eval/harness_command.py:36

    if not isinstance(block, dict):
        raise ValueError("run request is missing a 'benchmark' block")
    return block


def dataset_args(benchmark: dict[str, Any]) -> list[str]:
    dataset = benchmark.get("dataset") or {}
    kind = dataset.get("kind")
    if kind == benchmarks.DATASET_REGISTRY:
        name = dataset.get("name")
        if not name:
            raise ValueError("registry dataset requires a name")
        return ["-d", name]
    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
        data_dir = dataset.get("data_dir")
        if not data_dir:
            raise ValueError("path dataset requires data_dir")
        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")


def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
    """Where the controller clones the dataset repo for path datasets."""
    return f"/tmp/datasets/{benchmark['id']}"


def dataset_path(benchmark: dict[str, Any]) -> str:
    """Repo-relative task directory inside the cloned dataset repo."""
    dataset = benchmark.get("dataset") or {}
    return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"


def harness_binary(benchmark: dict[str, Any]) -> str:
    harness = benchmark.get("harness")
    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):
        raise ValueError(f"unsupported harness: {harness}")
    return harness

View on GitHub (pinned to f4178619ac)