{"record":{"id":"dfc4d1c83a71de21","repo":"zed-industries/zed","slug":"unsupported-dataset-kind-kind","errorCode":null,"errorMessage":"unsupported dataset kind: {kind}","messagePattern":"unsupported dataset kind: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":36,"sourceCode":"    if not isinstance(block, dict):\n        raise ValueError(\"run request is missing a 'benchmark' block\")\n    return block\n\n\ndef dataset_args(benchmark: dict[str, Any]) -> list[str]:\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind == benchmarks.DATASET_REGISTRY:\n        name = dataset.get(\"name\")\n        if not name:\n            raise ValueError(\"registry dataset requires a name\")\n        return [\"-d\", name]\n    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        data_dir = dataset.get(\"data_dir\")\n        if not data_dir:\n            raise ValueError(\"path dataset requires data_dir\")\n        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n\n\ndef dataset_clone_dir(benchmark: dict[str, Any]) -> str:\n    \"\"\"Where the controller clones the dataset repo for path datasets.\"\"\"\n    return f\"/tmp/datasets/{benchmark['id']}\"\n\n\ndef dataset_path(benchmark: dict[str, Any]) -> str:\n    \"\"\"Repo-relative task directory inside the cloned dataset repo.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    return f\"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}\"\n\n\ndef harness_binary(benchmark: dict[str, Any]) -> str:\n    harness = benchmark.get(\"harness\")\n    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):\n        raise ValueError(f\"unsupported harness: {harness}\")\n    return harness","sourceCodeStart":18,"sourceCodeEnd":54,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/harness_command.py#L18-L54","documentation":"Raised by harness_command.dataset_args() when dataset.kind is neither \"registry\", \"path\", nor \"pier_path\" — including when the dataset block is absent entirely (kind is None). These three kinds are the only provisioning strategies the controller knows: hub pull (-d), git clone for Harbor (-p), git clone for Pier (-p under pier). The kind constants live in benchmarks.py:25-27.","triggerScenarios":"Spelling mistakes in kind (\"Registry\", \"git\", \"local\", \"pier-path\"); a run request produced by a newer CLI that added a new dataset kind, replayed on an older deployed controller; a benchmark block whose dataset key was dropped so dataset defaults to {} and kind is None.","commonSituations":"Version skew between the local CLI and the deployed Modal app after new dataset kinds are introduced; hand-rolled benchmark metadata; renaming kind values in a fork without redeploying the controller image.","solutions":["Set dataset.kind to exactly \"registry\", \"path\", or \"pier_path\".","Regenerate the benchmark block from the registry (benchmarks.benchmark_metadata) matching the zed_eval version you deploy.","Redeploy the Modal app (zed-eval deploy) so the controller's harness_command knows the same kinds as your client."],"exampleFix":"# before\n\"dataset\": {\"kind\": \"hub\", \"name\": \"scale-ai/swe-atlas-qna\"}\n# -> ValueError: unsupported dataset kind: hub\n\n# after\n\"dataset\": {\"kind\": \"registry\", \"name\": \"scale-ai/swe-atlas-qna\"}","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\nVALID_KINDS = {\n    benchmarks.DATASET_REGISTRY,\n    benchmarks.DATASET_PATH,\n    benchmarks.DATASET_PIER_PATH,\n}\nif dataset.get(\"kind\") not in VALID_KINDS:\n    raise SystemExit(\n        f\"dataset.kind must be one of {sorted(VALID_KINDS)}, got {dataset.get('kind')!r}\"\n    )","typeGuard":"def is_known_dataset_kind(kind: object) -> bool:\n    return kind in (\"registry\", \"path\", \"pier_path\")","tryCatchPattern":"try:\n    args_ = harness_command.dataset_args(benchmark)\nexcept ValueError as error:\n    raise SystemExit(str(error))  # echo kind so the mismatch is obvious","preventionTips":["Keep client CLI and deployed Modal app on the same zed_eval version to avoid kind skew.","Never hand-type kind strings; use the benchmarks.DATASET_* constants.","Redeploy (zed-eval deploy) after any registry change that adds dataset kinds."],"tags":["validation","dataset","version-skew"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}