{"record":{"id":"9de8ac09b2adad58","repo":"zed-industries/zed","slug":"unsupported-harness-harness","errorCode":null,"errorMessage":"unsupported harness: {harness}","messagePattern":"unsupported harness: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/harness_command.py","lineNumber":53,"sourceCode":"        return [\"-p\", dataset_path(benchmark)]\n    raise ValueError(f\"unsupported dataset kind: {kind}\")\n\n\ndef dataset_clone_dir(benchmark: dict[str, Any]) -> str:\n    \"\"\"Where the controller clones the dataset repo for path datasets.\"\"\"\n    return f\"/tmp/datasets/{benchmark['id']}\"\n\n\ndef dataset_path(benchmark: dict[str, Any]) -> str:\n    \"\"\"Repo-relative task directory inside the cloned dataset repo.\"\"\"\n    dataset = benchmark.get(\"dataset\") or {}\n    return f\"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}\"\n\n\ndef harness_binary(benchmark: dict[str, Any]) -> str:\n    harness = benchmark.get(\"harness\")\n    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):\n        raise ValueError(f\"unsupported harness: {harness}\")\n    return harness\n\n\ndef eval_cli_timeout(run_request: dict[str, Any], benchmark: dict[str, Any]) -> int:\n    return int(\n        run_request.get(\"eval_cli_timeout\")\n        or benchmark.get(\"default_timeout_secs\")\n        or config.DEFAULT_SANDBOX_TIMEOUT_SECS\n    )\n\n\ndef build_harness_command(run_request: dict[str, Any], jobs_dir: str) -> list[str]:\n    benchmark = _benchmark_block(run_request)\n    build_id = run_request.get(\"build_id\")\n    if not build_id:\n        raise ValueError(\"zed benchmarks require build_id\")\n    volume_name = run_request[\"volume_name\"]\n    api_secret_name = run_request[\"api_secret_name\"]","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/zed-industries/zed/blob/f4178619acd0d47ea1f76a2025c42962c6d6638c/crates/eval_cli/zed_eval/harness_command.py#L35-L71","documentation":"Raised by harness_binary() in zed_eval/harness_command.py when benchmark['harness'] is neither benchmarks.HARNESS_HARBOR ('harbor') nor benchmarks.HARNESS_PIER ('pier'). The returned string is used both as the executable name at the head of the harness command and to pick the agent import path (zed_eval.pier_agent vs zed_eval.agent), so an unknown harness cannot be dispatched.","triggerScenarios":"build_harness_command() reads benchmark['harness'] from the run request block and finds a typo ('Harbor', 'pier '), None, or a new harness name. Happens when a Benchmark is registered with an invented harness string, or a benchmark block is constructed/edited outside benchmarks.py.","commonSituations":"Onboarding a third harness (e.g. a new Harbor fork) and registering it in benchmarks.py without updating harness_binary(); whitespace/case typos in hand-written benchmark metadata; replaying a run request JSON whose harness field was edited; copy-pasting a Benchmark entry and changing harness to an unregistered value.","solutions":["Set harness to benchmarks.HARNESS_HARBOR or benchmarks.HARNESS_PIER (the constants, not raw strings) in the Benchmark registration","If you added a new harness, extend the membership check in harness_binary() and the agent-import-path selection in build_harness_command() so the new harness gets its own agent module","If the field is None, the benchmark block lost its 'harness' key — rebuild the block with benchmarks.benchmark_metadata()","Sanitize user-supplied harness values with .strip().lower() before they reach the registry"],"exampleFix":"# before\n\"my-bench\": Benchmark(id=\"my-bench\", harness=\"Harbor\", ...)\n# after\n\"my-bench\": Benchmark(id=\"my-bench\", harness=benchmarks.HARNESS_HARBOR, ...)","handlingStrategy":"validation","validationCode":"from zed_eval import benchmarks\n\nharness = run_request[\"benchmark\"].get(\"harness\")\nif harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):\n    raise SystemExit(\n        f\"harness must be {benchmarks.HARNESS_HARBOR!r} or \"\n        f\"{benchmarks.HARNESS_PIER!r}, got {harness!r}\"\n    )","typeGuard":"from zed_eval import benchmarks\n\ndef is_supported_harness(benchmark: dict) -> bool:\n    harness = benchmark.get(\"harness\")\n    return harness in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER)","tryCatchPattern":null,"preventionTips":["Register harness via the HARNESS_HARBOR/HARNESS_PIER constants, never literals","Normalize external harness input with .strip().lower() before it reaches the registry","Keep harness_binary()'s tuple and build_harness_command()'s agent-import-path selection in sync when adding a harness"],"tags":["python","zed-eval","harness","enum-validation","benchmark-config"],"backgroundTag":"unsupported-config-value","analyzedSha":"f4178619acd0d47ea1f76a2025c42962c6d6638c","analyzedAt":"2026-08-20T19:29:52.058Z","contentChangedAt":"2026-08-20T19:29:52.058Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}