zed-industries/zed · error · ValueError

unsupported harness: {harness}

Error message

unsupported harness: {harness}

What it means

Raised by harness_command.harness_binary() when benchmark.harness is not "harbor" or "pier". The value doubles as the executable name placed first on the harness command line, and selects the installed-agent interface (zed_eval.agent:ZedAgent vs zed_eval.pier_agent:ZedPierAgent). Pier is a Harbor fork used for air-gapped benchmarks like DeepSWE that need per-agent network allowlists.

Source

Thrown at crates/eval_cli/zed_eval/harness_command.py:53

        return ["-p", dataset_path(benchmark)]
    raise ValueError(f"unsupported dataset kind: {kind}")


def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
    """Where the controller clones the dataset repo for path datasets."""
    return f"/tmp/datasets/{benchmark['id']}"


def dataset_path(benchmark: dict[str, Any]) -> str:
    """Repo-relative task directory inside the cloned dataset repo."""
    dataset = benchmark.get("dataset") or {}
    return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"


def harness_binary(benchmark: dict[str, Any]) -> str:
    harness = benchmark.get("harness")
    if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):
        raise ValueError(f"unsupported harness: {harness}")
    return harness


def eval_cli_timeout(run_request: dict[str, Any], benchmark: dict[str, Any]) -> int:
    return int(
        run_request.get("eval_cli_timeout")
        or benchmark.get("default_timeout_secs")
        or config.DEFAULT_SANDBOX_TIMEOUT_SECS
    )


def build_harness_command(run_request: dict[str, Any], jobs_dir: str) -> list[str]:
    benchmark = _benchmark_block(run_request)
    build_id = run_request.get("build_id")
    if not build_id:
        raise ValueError("zed benchmarks require build_id")
    volume_name = run_request["volume_name"]
    api_secret_name = run_request["api_secret_name"]

View on GitHub (pinned to bc538def45)

Solutions

1. Set benchmark.harness to exactly benchmarks.HARNESS_HARBOR ("harbor") or benchmarks.HARNESS_PIER ("pier").
2. Use Pier only when per-agent network allowlists are required (air-gapped tasks); otherwise use Harbor.
3. After changing benchmarks.py, redeploy the Modal app so the controller uses the updated registry.

Example fix

# before
Benchmark(id="mybench", harness="Harbor", ...)
# -> ValueError: unsupported harness: Harbor

# after
from zed_eval.benchmarks import HARNESS_HARBOR
Benchmark(id="mybench", harness=HARNESS_HARBOR, ...)

from zed_eval import benchmarks

harness = benchmark.get("harness")
if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):
    raise SystemExit("benchmark.harness must be 'harbor' or 'pier'")

def is_supported_harness(harness: object) -> bool:
    return harness in ("harbor", "pier")

try:
    binary = harness_command.harness_binary(benchmark)
except ValueError as error:
    raise SystemExit(str(error))

When it happens

Trigger: Hand-built benchmark blocks with harness spelled "Harbor", "harbor2", or missing (None); registering a new benchmark without setting Benchmark(harness=...); replaying a request against a controller that only knows the two harnesses while the block names a third.

Common situations: Adding a new benchmark family and forgetting the harness field; forks that add a harness variant without updating harness_binary's allowlist; case mismatches since the comparison is exact.

Related errors

• unknown judge preset '{name}' (valid: {valid})
• registry dataset requires a name
• path dataset requires data_dir
• unknown benchmark '{benchmark_id}' (valid: {valid})
• unknown benchmark '{selector}' (valid: {valid})

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/9de8ac09b2adad58. Report an issue: GitHub.