{"record":{"id":"4c4b6b4765db4927","repo":"abhigyanpatwari/GitNexus","slug":"candidate-overlay-changed-during-the-benchmark-run","errorCode":null,"errorMessage":"candidate overlay changed during the benchmark run","messagePattern":"candidate overlay changed during the benchmark run","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"eval/workflow_bench/runner.py","lineNumber":1153,"sourceCode":"                                if not setup.ok:\n                                    raise ManagedProcessError(setup_command, setup)\n                            # Tamper-evidence: setup must not have rewritten the base\n                            # skills, verified before any candidate overlay lands.\n                            require_skill_fingerprint(\n                                worktree,\n                                execution_arm,\n                                base_skill_digest,\n                                phase=\"task setup\",\n                            )\n                            if arm in CANDIDATE_ARMS:\n                                assert candidate_overlay is not None\n                                applied_digest = apply_candidate_overlay(\n                                    candidate_overlay,\n                                    worktree,\n                                    sandbox=sandbox,\n                                )\n                                if applied_digest != overlay_digest:\n                                    raise RuntimeError(\"candidate overlay changed during the benchmark run\")\n                            # The digest the model must preserve during its run is the\n                            # post-overlay skill surface (candidate skills for\n                            # candidate arms; unchanged base skills otherwise).\n                            expected_skill_digest = skill_fingerprint(worktree, execution_arm)\n                            orig_sha = _sandbox_git(sandbox, [\"rev-parse\", \"HEAD\"]).strip()\n                            if not re.fullmatch(r\"[0-9a-fA-F]{40,64}\", orig_sha):\n                                raise RuntimeError(\"sandboxed candidate setup did not produce an immutable commit\")\n                            before_work_digest = (\n                                implementation_diff_digest(sandbox, orig_sha)\n                                if execution_arm in IMPLEMENTATION_ARMS\n                                else \"\"\n                            )\n                            record = run_arm(\n                                execution_arm,\n                                task,\n                                worktree,\n                                args,\n                                sandbox=sandbox,","sourceCodeStart":1135,"sourceCodeEnd":1171,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/runner.py#L1135-L1171","documentation":"Raised after apply_candidate_overlay() returns a digest that does not match the overlay_digest computed once at program start (runner.py:959). The overlay digest is a content hash of the candidate skill overlay directory captured before any arm runs; the harness re-derives it after applying the overlay into the worktree to detect that the overlay's bytes drifted mid-run. This is a tamper-evidence/integrity guard: a mismatch means the candidate skill surface the model actually saw differs from the surface that was approved at startup.","triggerScenarios":"The harness calls candidate_overlay_digest(candidate_overlay) at startup (runner.py:959) and apply_candidate_overlay(candidate_overlay, worktree, sandbox=sandbox) at runner.py:1147 during a candidate arm run; if the returned applied_digest != overlay_digest the RuntimeError fires. Concretely: a background process or second benchmark instance writes to the --candidate-overlay directory between those two calls, the overlay directory contains a symlink whose target changes, or candidate_overlay_payload() is non-deterministic (e.g. it picks up editor swap files or .DS_Store that vary).","commonSituations":"Running two evolve.py/runner.py processes pointed at the same --candidate-overlay scratch dir; an editor auto-saving into the overlay tree; the overlay living in a synced (Dropbox/icloud) folder that mutates during the run; regenerating the overlay mid-benchmark.","solutions":["Stop any concurrent writer to the --candidate-overlay path and re-run the benchmark from a clean overlay snapshot copied to a stable local directory.","Verify candidate_overlay_digest() is deterministic for your overlay by computing it twice before launching the run; remove non-reproducible files (swap files, .DS_Store, editor backups) from the overlay tree.","Pin the overlay to a read-only copy (chmod -R a-w or a git commit SHA) for the duration of the run so nothing can mutate it."],"exampleFix":"# before: pass a live, mutable overlay dir\nrunner.py --candidate-overlay /tmp/overlay-wip --arms candidate_workflow workflow\n\n# after: snapshot the overlay to a frozen copy first\ncp -a /tmp/overlay-wip /tmp/overlay-frozen && chmod -R a-w /tmp/overlay-frozen\nrunner.py --candidate-overlay /tmp/overlay-frozen --arms candidate_workflow workflow","handlingStrategy":"validation","validationCode":"from eval.workflow_bench.evolution import candidate_overlay_digest\n\n# Compute the overlay digest twice before launching the benchmark;\n# any difference means a concurrent writer is active.\nd1 = candidate_overlay_digest(overlay_path)\nimport time; time.sleep(0.1)\nd2 = candidate_overlay_digest(overlay_path)\nassert d1 == d2, f\"overlay is being mutated mid-read: {d1} vs {d2}\"\n# Freeze the overlay so no further mutation is possible\nimport shutil, os\nfrozen = \"/tmp/overlay-frozen\"\nshutil.copytree(overlay_path, frozen, dirs_exist_ok=True)\nfor root, dirs, files in os.walk(frozen):\n    os.chmod(root, 0o555)\n    for f in files:\n        os.chmod(os.path.join(root, f), 0o444)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never point --candidate-overlay at a directory a live editor or sync client can write to.","Copy the overlay to a read-only location and pass that path to the harness.","Run a single benchmark instance per overlay directory; serialize concurrent runs."],"tags":["integrity","candidate-overlay","benchmark","tamper-evidence"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}