{"record":{"id":"dfc4d1c83a71de21","repo":"zed-industries/zed","slug":"unsupported-dataset-kind-kind","errorCode":null,"errorMessage":"unsupported dataset kind: {kind}","messagePattern":"unsupported dataset kind: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":36,"sourceCode":"    if not isinstance(block, dict):\n        raise ValueError(\"run request is missing a 'benchmark' block\")\n    return block\n\n\ndef dataset_args(benchmark: dict[str, Any]) -> list[str]:\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind == benchmarks.DATASET_REGISTRY:\n        name = dataset.get(\"name\")\n        if not name:\n            raise ValueError(\"registry dataset requires a name\")\n        return [\"-d\", name]\n    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        data_dir = dataset.get(\"data_dir\")\n        if not data_dir:\n            raise ValueError(\"path dataset requires data_dir\")\n        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n\n\ndef dataset_clone_dir(benchmark: dict[str, Any]) -> str:\n    \"\"\"Where the controller clones the dataset repo for path datasets.\"\"\"\n    return f\"/tmp/datasets/{benchmark['id']}\"\n\n\ndef dataset_path(benchmark: dict[str, Any]) -> str:\n    \"\"\"Repo-relative task directory inside the cloned dataset repo.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    return f\"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}\"\n\n\ndef harness_binary(benchmark: dict[str, Any]) -> str:\n    harness = benchmark.get(\"harness\")\n    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):\n        raise ValueError(f\"unsupported harness: {harness}\")\n    return harness","sourceCodeStart":18,"sourceCodeEnd":54,"githubUrl":"https://github.com/zed-industries/zed/blob/f4178619acd0d47ea1f76a2025c42962c6d6638c/crates/eval_cli/zed_eval/harness_command.py#L18-L54","documentation":"Raised by dataset_args() in zed_eval/harness_command.py when the benchmark's dataset kind is none of the three supported constants: 'registry', 'path', or 'pier_path' (benchmarks.DATASET_REGISTRY/DATASET_PATH/DATASET_PIER_PATH). The function maps kind to a CLI flag (-d <name> for registry, -p <dir> for path kinds); any other value has no mapping, so it refuses rather than guessing.","triggerScenarios":"run_request['benchmark']['dataset']['kind'] is missing (None), misspelled ('Path', 'local', 'git'), or a genuinely new kind added to benchmarks.py without extending dataset_args(). Also triggered by hand-crafted or JSON-loaded benchmark blocks that bypass benchmarks.benchmark_metadata().","commonSituations":"Adding a new dataset provisioning mode (e.g. s3-backed) and forgetting the branch in dataset_args; typos when a benchmark block is edited as raw JSON; older run requests replayed against a newer zed_eval whose kind vocabulary changed; test fixtures inventing ad-hoc kinds.","solutions":["Set dataset kind to one of the exact strings 'registry', 'path', or 'pier_path' — preferably via the benchmarks.DATASET_* constants so typos cannot compile/lint past you","If you introduced a new kind in benchmarks.py, add the corresponding branch to dataset_args() in harness_command.py returning the right -d/-p args","If the block came from a stored run request JSON, fix the kind at the source and regenerate the request via benchmark_metadata()","If kind is None, the whole dataset sub-block was dropped — restore it (benchmark.get('dataset') or {} masks the absence until this check fires)"],"exampleFix":"# before\nbenchmark[\"dataset\"] = {\"kind\": \"local\", \"data_dir\": \"data/tw\"}\n# after\nfrom . import benchmarks\nbenchmark[\"dataset\"] = {\n    \"kind\": benchmarks.DATASET_PATH,\n    \"data_dir\": \"data/tw\",\n}","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\nVALID_KINDS = {\n    benchmarks.DATASET_REGISTRY,\n    benchmarks.DATASET_PATH,\n    benchmarks.DATASET_PIER_PATH,\n}\n\nkind = (run_request[\"benchmark\"].get(\"dataset\") or {}).get(\"kind\")\nif kind not in VALID_KINDS:\n    raise SystemExit(\n        f\"dataset.kind must be one of {sorted(VALID_KINDS)}, got {kind!r}\"\n    )","typeGuard":"from zed_eval import benchmarks\n\ndef is_known_dataset_kind(benchmark: dict) -> bool:\n    kind = (benchmark.get(\"dataset\") or {}).get(\"kind\")\n    return kind in (\n        benchmarks.DATASET_REGISTRY,\n        benchmarks.DATASET_PATH,\n        benchmarks.DATASET_PIER_PATH,\n    )","tryCatchPattern":"try:\n    harness_command.dataset_args(benchmark)\nexcept ValueError as error:\n    # error message names the unsupported kind — surface it to the operator\n    log(f\"rejecting benchmark {benchmark.get('id')}: {error}\")\n    return","preventionTips":["Use the benchmarks.DATASET_* constants everywhere instead of raw strings","When adding a dataset kind, update dataset_args() in the same commit and add a registry test iterating all BENCHMARKS through it","Validate run-request JSON against the known kind vocabulary before spawning controllers"],"tags":["python","zed-eval","dataset-kind","enum-validation","harness"],"backgroundTag":"unsupported-config-value","analyzedSha":"f4178619acd0d47ea1f76a2025c42962c6d6638c","analyzedAt":"2026-08-20T19:29:52.058Z","contentChangedAt":"2026-08-20T19:29:52.058Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}