abhigyanpatwari/GitNexus · critical · SystemExit

[harness-health] incumbent arm(s) {broken_incumbents} resolv

Error message

[harness-health] incumbent arm(s) {broken_incumbents} resolved zero tasks across every valid run — this looks like an environment/harness failure, not a normal candidate miss. See the errors column in report.md and error_detail in results.jsonl. Exiting non-zero rather than reporting a quiet no-promotion.

What it means

Raised at end-of-run (runner.py:1384) when broken_incumbent_arms() reports that every incumbent (currently-shipped) skill arm resolved zero tasks across all its runs. Incumbents are presumed-working baselines; resolving nothing across the board signals an environment or harness failure (missing interpreter, stale skill fingerprint, sandbox misconfig) rather than a legitimate candidate miss. The harness exits non-zero instead of emitting a misleading 'no promotion, incumbent stands' verdict.

Source

Thrown at eval/workflow_bench/runner.py:1384

                    min_improvement_pct=args.promotion_min_improvement,
                    max_task_regression_pct=args.promotion_max_task_regression,
                )
                for candidate_arm in candidate_arms
            ],
        }
        (out_dir / "promotion.json").write_text(json.dumps(promotion, indent=2) + "\n")
    print(f"\n{report}\n\nWritten to {out_dir}/")
    broken_incumbents = broken_incumbent_arms(results, set(CANDIDATE_ARMS.values()))
    if broken_incumbents:
        # Fail loudly rather than let a broken environment read as a quiet
        # "no promotion, incumbent stands."
        print(
            f"[harness-health] incumbent arm(s) {', '.join(broken_incumbents)} resolved zero "
            "tasks across every valid run — this looks like an environment/harness failure, "
            "not a normal candidate miss. See the errors column in report.md and error_detail "
            "in results.jsonl. Exiting non-zero rather than reporting a quiet no-promotion."
        )
        raise SystemExit(1)
    if outage_tripped:
        # Non-zero exit so a driver (evolve.py) treats the partial benchmark as a
        # failed run and halts instead of proposing from outage-truncated evidence.
        raise SystemExit(1)


if __name__ == "__main__":
    main()

View on GitHub (pinned to d540b00184)

Solutions

Open report.md and the errors column, then results.jsonl error_detail for the incumbent arm to find the dominant error_kind (e.g. managed-process, evidence-unverified).
Run the incumbent arm alone for one task with verbose logging to reproduce: confirm the claude binary launches, the sandbox can exec node/python, and the skill files are readable.
Re-run preflight: ensure preflight_bubblewrap() and require_claude_sandbox_helpers() pass and that --claude-bin points at a working claude executable.

Defensive patterns

Strategy: validation

Validate before calling

import subprocess, shutil, os

# Preflight the incumbent path: the claude binary runs and skills are readable.
claude = shutil.which("claude") or os.environ.get("CLAUDE_BIN")
assert claude and os.access(claude, os.X_OK), f"claude binary not executable: {claude}"

skills = pathlib.Path(".claude/skills")
assert skills.is_dir(), "missing .claude/skills — incumbent arm cannot load"
for md in skills.rglob("SKILL.md"):
    assert os.access(md, os.R_OK), f"unreadable skill file: {md}"

# Smoke-run one incumbent task and assert resolved > 0 before the full bench.
proc = subprocess.run(
    ["python", "-m", "eval.workflow_bench.runner",
     "--arms", "workflow", "--runs", "1", "--tasks", "<smoke-task>"],
    capture_output=True, text=True,
)
assert proc.returncode == 0, f"incumbent smoke failed:\n{proc.stderr}"

Try / catch

try:
    main()
except SystemExit as exc:
    if exc.code == 1 and "harness-health" in (last_printed_line or ""):
        # Do NOT auto-retry; inspect report.md/results.jsonl for the dominant
        # incumbent error_kind and fix the environment before re-running.
        log.error("incumbent arms broken — fix environment, see report.md")
        raise

Prevention

Run a single-task incumbent smoke test before the full benchmark.
Ensure the claude CLI, node, and python are on PATH and executable inside the sandbox.
Validate .claude/skills is a readable directory in every worktree before launching arms.

When it happens

Trigger: broken_incumbent_arms(results, set(CANDIDATE_ARMS.values())) at runner.py:1374 returns a non-empty list: an incumbent arm present in results has resolved == 0 for every task it ran. Caused by: the trusted claude binary missing or non-executable, required node/python toolchain absent in the sandbox, all skill SKILL.md files unreadable (permission/format), or every run hitting the outage-streak breaker.

Common situations: Running the benchmark on a fresh CI runner without the claude CLI installed; sandbox bubblewrap (bwrap) blocking the interpreter path; a repo-wide chmod that made .claude/skills unreadable; API key/credential missing so every session errors before doing work; wrong --claude-bin path.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/8b19a45ca9d8fead. Report an issue: GitHub.