{"record":{"id":"b025e44facb14f60","repo":"zed-industries/zed","slug":"benchmark-benchmark-id-path-dataset-requires","errorCode":null,"errorMessage":"benchmark {benchmark['id']} path dataset requires repo_url","messagePattern":"benchmark (.+?) path dataset requires repo_url","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/modal_app.py","lineNumber":111,"sourceCode":"        \"echo 'WARNING: pier install failed; DeepSWE runs will not work'\",\n    )\n)\n\n\ndef provision_benchmark_dataset(run_request: dict[str, Any], log: Any) -> None:\n    \"\"\"Clone a path-backed benchmark dataset (SWE-Atlas tw, DeepSWE) into the\n    location the harness command expects. Registry datasets need no provisioning;\n    the harness pulls them from its hub.\"\"\"\n    benchmark = run_request[\"benchmark\"]\n    dataset = benchmark.get(\"dataset\") or {}\n    kind = dataset.get(\"kind\")\n    if kind not in (\"path\", \"pier_path\"):\n        return\n\n    repo_url = dataset.get(\"repo_url\")\n    repo_ref = dataset.get(\"repo_ref\") or \"main\"\n    if not repo_url:\n        raise ValueError(f\"benchmark {benchmark['id']} path dataset requires repo_url\")\n\n    clone_dir = pathlib.Path(harness_command.dataset_clone_dir(benchmark))\n    if clone_dir.exists():\n        shutil.rmtree(clone_dir)\n    clone_dir.mkdir(parents=True, exist_ok=True)\n    log(f\"Fetching {benchmark['id']} dataset {repo_url}@{repo_ref}\")\n    subprocess.run([\"git\", \"init\", \"-q\", str(clone_dir)], check=True)\n    subprocess.run(\n        [\"git\", \"fetch\", \"--depth\", \"1\", repo_url, repo_ref],\n        cwd=clone_dir,\n        check=True,\n    )\n    subprocess.run([\"git\", \"checkout\", \"-q\", \"FETCH_HEAD\"], cwd=clone_dir, check=True)\n    data_dir = pathlib.Path(harness_command.dataset_path(benchmark))\n    if not data_dir.exists():\n        raise FileNotFoundError(f\"benchmark dataset directory not found: {data_dir}\")\n\n","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/modal_app.py#L93-L129","documentation":"Raised by modal_app.provision_benchmark_dataset(), running inside the Modal controller, when a benchmark's dataset.kind is \"path\" or \"pier_path\" but dataset.repo_url is missing. Path datasets are provisioned by git init + git fetch --depth 1 of repo_url at repo_ref (default \"main\") into /tmp/datasets/{id}, so a clone URL is mandatory. Registry datasets return early and never hit this. All in-tree path benchmarks (swe-atlas-tw, deepswe) define repo_url in benchmarks.py.","triggerScenarios":"Registering a custom path/pier_path benchmark whose DatasetRef omits repo_url; hand-editing benchmark metadata and dropping repo_url; a run request whose embedded benchmark block was built against a modified registry. The failure happens remotely after the controller has already started and (if applicable) waited for the build.","commonSituations":"Adding a new git-backed benchmark and forgetting the URL; assuming repo_url is optional because repo_ref has a default; forking the registry and renaming fields without redeploying.","solutions":["Set dataset.repo_url to a fetchable https git URL (e.g. https://github.com/datacurve-ai/deep-swe.git) in the benchmark's DatasetRef.","Redeploy the Modal app after editing benchmarks.py so the controller sees the fixed metadata, then relaunch.","If the tasks are on the harness hub, use kind=\"registry\" with name and no provisioning is needed."],"exampleFix":"# before\nDatasetRef(kind=DATASET_PATH, data_dir=\"data/tw\")  # no repo_url\n# controller: ValueError: benchmark swe-atlas-tw path dataset requires repo_url\n\n# after\nDatasetRef(\n    kind=DATASET_PATH,\n    repo_url=SWE_ATLAS_REPO_URL,\n    repo_ref=SWE_ATLAS_REPO_REF,\n    data_dir=\"data/tw\",\n)","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\nfor benchmark in benchmarks.BENCHMARKS.values():\n    dataset = benchmark.dataset\n    if dataset.kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):\n        assert dataset.repo_url, f\"{benchmark.id} path dataset missing repo_url\"\n        assert dataset.data_dir, f\"{benchmark.id} path dataset missing data_dir\"","typeGuard":"def is_provisionable_dataset(dataset: dict) -> bool:\n    kind = dataset.get(\"kind\")\n    if kind in (\"path\", \"pier_path\"):\n        return bool(dataset.get(\"repo_url\"))\n    return kind == \"registry\"","tryCatchPattern":null,"preventionTips":["Add a registry unit test: every path/pier_path benchmark defines repo_url (and data_dir).","Validate benchmark metadata locally before launching; this error only surfaces remotely after the run starts.","Redeploy the Modal app after fixing benchmarks.py so controllers use the corrected registry."],"tags":["dataset","validation","modal","controller"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}