{"record":{"id":"9de8ac09b2adad58","repo":"zed-industries/zed","slug":"unsupported-harness-harness","errorCode":null,"errorMessage":"unsupported harness: {harness}","messagePattern":"unsupported harness: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":53,"sourceCode":"        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n\n\ndef dataset_clone_dir(benchmark: dict[str, Any]) -> str:\n    \"\"\"Where the controller clones the dataset repo for path datasets.\"\"\"\n    return f\"/tmp/datasets/{benchmark['id']}\"\n\n\ndef dataset_path(benchmark: dict[str, Any]) -> str:\n    \"\"\"Repo-relative task directory inside the cloned dataset repo.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    return f\"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}\"\n\n\ndef harness_binary(benchmark: dict[str, Any]) -> str:\n    harness = benchmark.get(\"harness\")\n    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):\n        raise ValueError(f\"unsupported harness: {harness}\")\n    return harness\n\n\ndef eval_cli_timeout(run_request: dict[str, Any], benchmark: dict[str, Any]) -> int:\n    return int(\n        run_request.get(\"eval_cli_timeout\")\n        or benchmark.get(\"default_timeout_secs\")\n        or config.DEFAULT_SANDBOX_TIMEOUT_SECS\n    )\n\n\ndef build_harness_command(run_request: dict[str, Any], jobs_dir: str) -> list[str]:\n    benchmark = _benchmark_block(run_request)\n    build_id = run_request.get(\"build_id\")\n    if not build_id:\n        raise ValueError(\"zed benchmarks require build_id\")\n    volume_name = run_request[\"volume_name\"]\n    api_secret_name = run_request[\"api_secret_name\"]","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/harness_command.py#L35-L71","documentation":"Raised by harness_command.harness_binary() when benchmark.harness is not \"harbor\" or \"pier\". The value doubles as the executable name placed first on the harness command line, and selects the installed-agent interface (zed_eval.agent:ZedAgent vs zed_eval.pier_agent:ZedPierAgent). Pier is a Harbor fork used for air-gapped benchmarks like DeepSWE that need per-agent network allowlists.","triggerScenarios":"Hand-built benchmark blocks with harness spelled \"Harbor\", \"harbor2\", or missing (None); registering a new benchmark without setting Benchmark(harness=...); replaying a request against a controller that only knows the two harnesses while the block names a third.","commonSituations":"Adding a new benchmark family and forgetting the harness field; forks that add a harness variant without updating harness_binary's allowlist; case mismatches since the comparison is exact.","solutions":["Set benchmark.harness to exactly benchmarks.HARNESS_HARBOR (\"harbor\") or benchmarks.HARNESS_PIER (\"pier\").","Use Pier only when per-agent network allowlists are required (air-gapped tasks); otherwise use Harbor.","After changing benchmarks.py, redeploy the Modal app so the controller uses the updated registry."],"exampleFix":"# before\nBenchmark(id=\"mybench\", harness=\"Harbor\", ...)\n# -> ValueError: unsupported harness: Harbor\n\n# after\nfrom zed_eval.benchmarks import HARNESS_HARBOR\nBenchmark(id=\"mybench\", harness=HARNESS_HARBOR, ...)","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\nharness = benchmark.get(\"harness\")\nif harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):\n    raise SystemExit(\"benchmark.harness must be 'harbor' or 'pier'\")","typeGuard":"def is_supported_harness(harness: object) -> bool:\n    return harness in (\"harbor\", \"pier\")","tryCatchPattern":"try:\n    binary = harness_command.harness_binary(benchmark)\nexcept ValueError as error:\n    raise SystemExit(str(error))","preventionTips":["Use the HARNESS_HARBOR/HARNESS_PIER constants, not literals, when defining benchmarks.","Pier is only needed for air-gapped tasks requiring network allowlists (e.g. DeepSWE).","Validate benchmark blocks before spawning controllers; the check is cheap and local."],"tags":["validation","harness","config"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}