{"record":{"id":"8b19a45ca9d8fead","repo":"abhigyanpatwari/GitNexus","slug":"harness-health-incumbent-arm-s-broken-incumben","errorCode":null,"errorMessage":"[harness-health] incumbent arm(s) {broken_incumbents} resolved zero tasks across every valid run — this looks like an environment/harness failure, not a normal candidate miss. See the errors column in report.md and error_detail in results.jsonl. Exiting non-zero rather than reporting a quiet no-promotion.","messagePattern":"\\[harness-health\\] incumbent arm\\(s\\) (.+?) resolved zero tasks across every valid run — this looks like an environment/harness failure, not a normal candidate miss\\. See the errors column in report\\.md and error_detail in results\\.jsonl\\. Exiting non-zero rather than reporting a quiet no-promotion\\.","errorType":"console","errorClass":"SystemExit","httpStatus":null,"severity":"critical","filePath":"eval/workflow_bench/runner.py","lineNumber":1384,"sourceCode":"                    min_improvement_pct=args.promotion_min_improvement,\n                    max_task_regression_pct=args.promotion_max_task_regression,\n                )\n                for candidate_arm in candidate_arms\n            ],\n        }\n        (out_dir / \"promotion.json\").write_text(json.dumps(promotion, indent=2) + \"\\n\")\n    print(f\"\\n{report}\\n\\nWritten to {out_dir}/\")\n    broken_incumbents = broken_incumbent_arms(results, set(CANDIDATE_ARMS.values()))\n    if broken_incumbents:\n        # Fail loudly rather than let a broken environment read as a quiet\n        # \"no promotion, incumbent stands.\"\n        print(\n            f\"[harness-health] incumbent arm(s) {', '.join(broken_incumbents)} resolved zero \"\n            \"tasks across every valid run — this looks like an environment/harness failure, \"\n            \"not a normal candidate miss. See the errors column in report.md and error_detail \"\n            \"in results.jsonl. Exiting non-zero rather than reporting a quiet no-promotion.\"\n        )\n        raise SystemExit(1)\n    if outage_tripped:\n        # Non-zero exit so a driver (evolve.py) treats the partial benchmark as a\n        # failed run and halts instead of proposing from outage-truncated evidence.\n        raise SystemExit(1)\n\n\nif __name__ == \"__main__\":\n    main()\n","sourceCodeStart":1366,"sourceCodeEnd":1393,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/runner.py#L1366-L1393","documentation":"Raised at end-of-run (runner.py:1384) when broken_incumbent_arms() reports that every incumbent (currently-shipped) skill arm resolved zero tasks across all its runs. Incumbents are presumed-working baselines; resolving nothing across the board signals an environment or harness failure (missing interpreter, stale skill fingerprint, sandbox misconfig) rather than a legitimate candidate miss. The harness exits non-zero instead of emitting a misleading 'no promotion, incumbent stands' verdict.","triggerScenarios":"broken_incumbent_arms(results, set(CANDIDATE_ARMS.values())) at runner.py:1374 returns a non-empty list: an incumbent arm present in results has resolved == 0 for every task it ran. Caused by: the trusted claude binary missing or non-executable, required node/python toolchain absent in the sandbox, all skill SKILL.md files unreadable (permission/format), or every run hitting the outage-streak breaker.","commonSituations":"Running the benchmark on a fresh CI runner without the claude CLI installed; sandbox bubblewrap (bwrap) blocking the interpreter path; a repo-wide chmod that made .claude/skills unreadable; API key/credential missing so every session errors before doing work; wrong --claude-bin path.","solutions":["Open report.md and the errors column, then results.jsonl error_detail for the incumbent arm to find the dominant error_kind (e.g. managed-process, evidence-unverified).","Run the incumbent arm alone for one task with verbose logging to reproduce: confirm the claude binary launches, the sandbox can exec node/python, and the skill files are readable.","Re-run preflight: ensure preflight_bubblewrap() and require_claude_sandbox_helpers() pass and that --claude-bin points at a working claude executable."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"import subprocess, shutil, os\n\n# Preflight the incumbent path: the claude binary runs and skills are readable.\nclaude = shutil.which(\"claude\") or os.environ.get(\"CLAUDE_BIN\")\nassert claude and os.access(claude, os.X_OK), f\"claude binary not executable: {claude}\"\n\nskills = pathlib.Path(\".claude/skills\")\nassert skills.is_dir(), \"missing .claude/skills — incumbent arm cannot load\"\nfor md in skills.rglob(\"SKILL.md\"):\n    assert os.access(md, os.R_OK), f\"unreadable skill file: {md}\"\n\n# Smoke-run one incumbent task and assert resolved > 0 before the full bench.\nproc = subprocess.run(\n    [\"python\", \"-m\", \"eval.workflow_bench.runner\",\n     \"--arms\", \"workflow\", \"--runs\", \"1\", \"--tasks\", \"<smoke-task>\"],\n    capture_output=True, text=True,\n)\nassert proc.returncode == 0, f\"incumbent smoke failed:\\n{proc.stderr}\"","typeGuard":null,"tryCatchPattern":"try:\n    main()\nexcept SystemExit as exc:\n    if exc.code == 1 and \"harness-health\" in (last_printed_line or \"\"):\n        # Do NOT auto-retry; inspect report.md/results.jsonl for the dominant\n        # incumbent error_kind and fix the environment before re-running.\n        log.error(\"incumbent arms broken — fix environment, see report.md\")\n        raise","preventionTips":["Run a single-task incumbent smoke test before the full benchmark.","Ensure the claude CLI, node, and python are on PATH and executable inside the sandbox.","Validate .claude/skills is a readable directory in every worktree before launching arms."],"tags":["harness-health","incumbent","benchmark","environment"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}