zed-industries/zed · error · ValueError

unsupported harness

Error message

unsupported harness: {harness}

What it means

Raised by harness_binary() in zed_eval/harness_command.py when benchmark['harness'] is neither benchmarks.HARNESS_HARBOR ('harbor') nor benchmarks.HARNESS_PIER ('pier'). The returned string is used both as the executable name at the head of the harness command and to pick the agent import path (zed_eval.pier_agent vs zed_eval.agent), so an unknown harness cannot be dispatched.

Solutions

1. Set harness to benchmarks.HARNESS_HARBOR or benchmarks.HARNESS_PIER (the constants, not raw strings) in the Benchmark registration
2. If you added a new harness, extend the membership check in harness_binary() and the agent-import-path selection in build_harness_command() so the new harness gets its own agent module
3. If the field is None, the benchmark block lost its 'harness' key — rebuild the block with benchmarks.benchmark_metadata()
4. Sanitize user-supplied harness values with .strip().lower() before they reach the registry

Example fix

# before
"my-bench": Benchmark(id="my-bench", harness="Harbor", ...)
# after
"my-bench": Benchmark(id="my-bench", harness=benchmarks.HARNESS_HARBOR, ...)

from zed_eval import benchmarks

harness = run_request["benchmark"].get("harness")
if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):
    raise SystemExit(
        f"harness must be {benchmarks.HARNESS_HARBOR!r} or "
        f"{benchmarks.HARNESS_PIER!r}, got {harness!r}"
    )

from zed_eval import benchmarks

def is_supported_harness(benchmark: dict) -> bool:
    harness = benchmark.get("harness")
    return harness in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER)

When it happens

Trigger: build_harness_command() reads benchmark['harness'] from the run request block and finds a typo ('Harbor', 'pier '), None, or a new harness name. Happens when a Benchmark is registered with an invented harness string, or a benchmark block is constructed/edited outside benchmarks.py.

Common situations: Onboarding a third harness (e.g. a new Harbor fork) and registering it in benchmarks.py without updating harness_binary(); whitespace/case typos in hand-written benchmark metadata; replaying a run request JSON whose harness field was edited; copy-pasting a Benchmark entry and changing harness to an unregistered value.

Related errors

• path dataset requires data_dir
• unsupported dataset kind
• debugpy installation failed (could not fetch Debugpy's…
• Failed to install or start judge proxy
• no fetched job at . Run `zed-eval fetch --experiment-name `…

AI-assisted analysis of zed-industries/zed@f4178619ac (2026-08-20). Data as JSON: /api/errors/9de8ac09b2adad58. Report an issue: GitHub.

Appendix: source

Thrown at crates/eval_cli/zed_eval/harness_command.py:53

        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")


def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
    """Where the controller clones the dataset repo for path datasets."""
    return f"/tmp/datasets/{benchmark['id']}"


def dataset_path(benchmark: dict[str, Any]) -> str:
    """Repo-relative task directory inside the cloned dataset repo."""
    dataset = benchmark.get("dataset") or {}
    return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"


def harness_binary(benchmark: dict[str, Any]) -> str:
    harness = benchmark.get("harness")
    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):
        raise ValueError(f"unsupported harness: {harness}")
    return harness


def eval_cli_timeout(run_request: dict[str, Any], benchmark: dict[str, Any]) -> int:
    return int(
        run_request.get("eval_cli_timeout")
        or benchmark.get("default_timeout_secs")
        or config.DEFAULT_SANDBOX_TIMEOUT_SECS
    )


def build_harness_command(run_request: dict[str, Any], jobs_dir: str) -> list[str]:
    benchmark = _benchmark_block(run_request)
    build_id = run_request.get("build_id")
    if not build_id:
        raise ValueError("zed benchmarks require build_id")
    volume_name = run_request["volume_name"]
    api_secret_name = run_request["api_secret_name"]

View on GitHub (pinned to f4178619ac)