{"record":{"id":"d695f9fbc9420c81","repo":"zed-industries/zed","slug":"registry-dataset-requires-a-name","errorCode":null,"errorMessage":"registry dataset requires a name","messagePattern":"registry dataset requires a name","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":29,"sourceCode":"from typing import Any\n\nfrom . import benchmarks, config\n\n\ndef _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:\n    block = run_request.get(\"benchmark\")\n    if not isinstance(block, dict):\n        raise ValueError(\"run request is missing a 'benchmark' block\")\n    return block\n\n\ndef dataset_args(benchmark: dict[str, Any]) -> list[str]:\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind == benchmarks.DATASET_REGISTRY:\n        name = dataset.get(\"name\")\n        if not name:\n            raise ValueError(\"registry dataset requires a name\")\n        return [\"-d\", name]\n    if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        data_dir = dataset.get(\"data_dir\")\n        if not data_dir:\n            raise ValueError(\"path dataset requires data_dir\")\n        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n\n\ndef dataset_clone_dir(benchmark: dict[str, Any]) -> str:\n    \"\"\"Where the controller clones the dataset repo for path datasets.\"\"\"\n    return f\"/tmp/datasets/{benchmark['id']}\"\n\n\ndef dataset_path(benchmark: dict[str, Any]) -> str:\n    \"\"\"Repo-relative task directory inside the cloned dataset repo.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    return f\"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}\"","sourceCodeStart":11,"sourceCodeEnd":47,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/harness_command.py#L11-L47","documentation":"Raised by harness_command.dataset_args() when the benchmark's dataset.kind == \"registry\" but dataset.name is missing or empty. Registry datasets are pulled from the harness hub via the -d <name> flag, so a hub dataset name is mandatory. All in-tree registry benchmarks (swe-atlas-qna, swe-atlas-rf, terminal-bench-2.1) carry a name; this fires only for hand-authored or mutated benchmark blocks.","triggerScenarios":"Building a benchmark block with kind=\"registry\" but no name key; copying benchmark metadata and deleting/rename-casing the name field; programmatic dataset blocks that set kind but rely on a separate variable for the hub name that is None.","commonSituations":"Adding a new benchmark to the registry and forgetting DatasetRef(name=...); typos like \"dataset_name\" instead of \"name\" when hand-writing the block; switching a benchmark from path to registry kind without adding the hub name.","solutions":["Set dataset.name to the harness hub dataset name, e.g. \"scale-ai/swe-atlas-qna\" or \"terminal-bench/terminal-bench-2-1\".","Prefer generating the block with benchmarks.benchmark_metadata() so the field layout is always correct.","If your tasks live in a git repo instead of the hub, use kind=\"path\"/\"pier_path\" with repo_url and data_dir rather than registry."],"exampleFix":"# before\n\"dataset\": {\"kind\": \"registry\"}  # -> ValueError: registry dataset requires a name\n\n# after\n\"dataset\": {\"kind\": \"registry\", \"name\": \"terminal-bench/terminal-bench-2-1\"}","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\ndataset = (run_request.get(\"benchmark\") or {}).get(\"dataset\") or {}\nif dataset.get(\"kind\") == benchmarks.DATASET_REGISTRY and not dataset.get(\"name\"):\n    raise SystemExit(\"registry dataset needs a hub 'name' (e.g. scale-ai/swe-atlas-qna)\")","typeGuard":"def is_valid_registry_dataset(dataset: dict) -> bool:\n    return dataset.get(\"kind\") == \"registry\" and bool(dataset.get(\"name\"))","tryCatchPattern":"try:\n    args_ = harness_command.dataset_args(benchmark)\nexcept ValueError as error:\n    raise SystemExit(f\"benchmark dataset block invalid: {error}\")","preventionTips":["Generate dataset blocks with benchmarks.DatasetRef + benchmark_metadata instead of hand-writing dicts.","Keep a unit test asserting every registry entry in BENCHMARKS has dataset.name set.","Remember: kind=registry means hub pull (-d name); git-backed data needs kind=path/pier_path."],"tags":["validation","dataset","config"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}