abhigyanpatwari/GitNexus · critical · RuntimeError

candidate overlay changed during the benchmark run

Error message

candidate overlay changed during the benchmark run

What it means

Raised after apply_candidate_overlay() returns a digest that does not match the overlay_digest computed once at program start (runner.py:959). The overlay digest is a content hash of the candidate skill overlay directory captured before any arm runs; the harness re-derives it after applying the overlay into the worktree to detect that the overlay's bytes drifted mid-run. This is a tamper-evidence/integrity guard: a mismatch means the candidate skill surface the model actually saw differs from the surface that was approved at startup.

Source

Thrown at eval/workflow_bench/runner.py:1153

                                if not setup.ok:
                                    raise ManagedProcessError(setup_command, setup)
                            # Tamper-evidence: setup must not have rewritten the base
                            # skills, verified before any candidate overlay lands.
                            require_skill_fingerprint(
                                worktree,
                                execution_arm,
                                base_skill_digest,
                                phase="task setup",
                            )
                            if arm in CANDIDATE_ARMS:
                                assert candidate_overlay is not None
                                applied_digest = apply_candidate_overlay(
                                    candidate_overlay,
                                    worktree,
                                    sandbox=sandbox,
                                )
                                if applied_digest != overlay_digest:
                                    raise RuntimeError("candidate overlay changed during the benchmark run")
                            # The digest the model must preserve during its run is the
                            # post-overlay skill surface (candidate skills for
                            # candidate arms; unchanged base skills otherwise).
                            expected_skill_digest = skill_fingerprint(worktree, execution_arm)
                            orig_sha = _sandbox_git(sandbox, ["rev-parse", "HEAD"]).strip()
                            if not re.fullmatch(r"[0-9a-fA-F]{40,64}", orig_sha):
                                raise RuntimeError("sandboxed candidate setup did not produce an immutable commit")
                            before_work_digest = (
                                implementation_diff_digest(sandbox, orig_sha)
                                if execution_arm in IMPLEMENTATION_ARMS
                                else ""
                            )
                            record = run_arm(
                                execution_arm,
                                task,
                                worktree,
                                args,
                                sandbox=sandbox,

View on GitHub (pinned to d540b00184)

Solutions

  1. Stop any concurrent writer to the --candidate-overlay path and re-run the benchmark from a clean overlay snapshot copied to a stable local directory.
  2. Verify candidate_overlay_digest() is deterministic for your overlay by computing it twice before launching the run; remove non-reproducible files (swap files, .DS_Store, editor backups) from the overlay tree.
  3. Pin the overlay to a read-only copy (chmod -R a-w or a git commit SHA) for the duration of the run so nothing can mutate it.

Example fix

# before: pass a live, mutable overlay dir
runner.py --candidate-overlay /tmp/overlay-wip --arms candidate_workflow workflow

# after: snapshot the overlay to a frozen copy first
cp -a /tmp/overlay-wip /tmp/overlay-frozen && chmod -R a-w /tmp/overlay-frozen
runner.py --candidate-overlay /tmp/overlay-frozen --arms candidate_workflow workflow
Defensive patterns

Strategy: validation

Validate before calling

from eval.workflow_bench.evolution import candidate_overlay_digest

# Compute the overlay digest twice before launching the benchmark;
# any difference means a concurrent writer is active.
d1 = candidate_overlay_digest(overlay_path)
import time; time.sleep(0.1)
d2 = candidate_overlay_digest(overlay_path)
assert d1 == d2, f"overlay is being mutated mid-read: {d1} vs {d2}"
# Freeze the overlay so no further mutation is possible
import shutil, os
frozen = "/tmp/overlay-frozen"
shutil.copytree(overlay_path, frozen, dirs_exist_ok=True)
for root, dirs, files in os.walk(frozen):
    os.chmod(root, 0o555)
    for f in files:
        os.chmod(os.path.join(root, f), 0o444)

Prevention

When it happens

Trigger: The harness calls candidate_overlay_digest(candidate_overlay) at startup (runner.py:959) and apply_candidate_overlay(candidate_overlay, worktree, sandbox=sandbox) at runner.py:1147 during a candidate arm run; if the returned applied_digest != overlay_digest the RuntimeError fires. Concretely: a background process or second benchmark instance writes to the --candidate-overlay directory between those two calls, the overlay directory contains a symlink whose target changes, or candidate_overlay_payload() is non-deterministic (e.g. it picks up editor swap files or .DS_Store that vary).

Common situations: Running two evolve.py/runner.py processes pointed at the same --candidate-overlay scratch dir; an editor auto-saving into the overlay tree; the overlay living in a synced (Dropbox/icloud) folder that mutates during the run; regenerating the overlay mid-benchmark.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/4c4b6b4765db4927. Report an issue: GitHub.