{"record":{"id":"4dab9ecd76cc6ad1","repo":"abhigyanpatwari/GitNexus","slug":"overlay-apply-failed-and-rollback-was-incomplete","errorCode":null,"errorMessage":"overlay apply failed and rollback was incomplete; recovery: {recovery}","messagePattern":"overlay apply failed and rollback was incomplete; recovery: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"eval/workflow_bench/promotion_apply.py","lineNumber":874,"sourceCode":"                    replacement[\"name\"],\n                )\n            except BaseException as rollback_exc:\n                if not rollback_exchange_is_valid(replacement, temporary_identity, displaced_state):\n                    rollback_failures.append(f\"{replacement['target']}: {type(rollback_exc).__name__}: {rollback_exc}\")\n            else:\n                if not rollback_exchange_is_valid(replacement, temporary_identity, displaced_state):\n                    rollback_failures.append(f\"{replacement['target']}: rollback parity check failed\")\n        if rollback_failures:\n            preserve_backups = True\n            recovery = _write_recovery_artifact(\n                root_descriptor,\n                repo_root,\n                failure=exc,\n                rollback_failures=rollback_failures,\n                replacements=replacements,\n                transaction_state=\"rollback-incomplete\",\n            )\n            raise RuntimeError(f\"overlay apply failed and rollback was incomplete; recovery: {recovery}\") from exc\n        rollback_complete = True\n        if isinstance(exc, (KeyboardInterrupt, SystemExit)):\n            raise\n        raise RuntimeError(\"overlay apply failed and all replacements were rolled back\") from exc\n    finally:\n        active_failure = sys.exc_info()[1]\n        try:\n            if not preserve_backups:\n                cleanup_failures: list[str] = []\n                cleanup_exception: BaseException | None = None\n                for replacement in replacements:\n                    for temporary in (replacement[\"candidate\"], replacement[\"backup\"]):\n                        if temporary is None:\n                            continue\n                        try:\n                            _unlink_temporary(replacement[\"parent_descriptor\"], temporary)\n                        except BaseException as cleanup_exc:\n                            cleanup_exception = cleanup_exc","sourceCodeStart":856,"sourceCodeEnd":892,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/promotion_apply.py#L856-L892","documentation":"Thrown from the `except` block of `apply_promoted_overlay` when the apply failed and one or more completed replacements could not be cleanly rolled back (`rollback_failures` non-empty). A recovery JSON artifact listing orphaned candidate/backup files is written to the repo root and its path is included in the message — manual intervention is required because the working tree is in an indeterminate state.","triggerScenarios":"Any failure (drift, parity, exception) during the apply loop after at least one exchange completed, where the rollback pass found that re-swapping a target did not restore the expected displaced state (see `rollback_exchange_is_valid` returning False at :683-687). The transaction is left 'rollback-incomplete' and a `.wfbench-overlay-recovery-*.json` is written.","commonSituations":"Filesystem that does not honor RENAME_EXCHANGE during rollback; concurrent writer corrupted a slot mid-rollback; storage failure; the original swap landed but the rollback swap raced with another edit. Whatever the cause, do NOT assume the tree is clean.","solutions":["Read the recovery JSON named in the message — it lists each target with `candidate_exists`/`backup_exists` so you can see which slots have orphaned temp files.","Inspect each named destination and decide manually whether the candidate or the backup represents the correct state; restore from `git` if unsure.","Delete the orphaned `.wfbench-overlay-recovery-*` candidate/backup files only after restoring destinations.","Fix the root cause (serialize writers / move to a reliable FS — see 411) before retrying apply."],"exampleFix":"# 1. open the recovery file named in the error\nimport json, pathlib\nrec = json.loads(pathlib.Path(recovery_path_in_message).read_text())\n# 2. for each record, restore the destination from git as the source of truth\nimport subprocess\nfor r in rec['backups']:\n    subprocess.run(['git', 'checkout', 'HEAD', '--', r['target']], check=True)\n# 3. remove orphaned candidate/backup temp files, then delete the recovery json","handlingStrategy":"try-catch","validationCode":"null","typeGuard":"null","tryCatchPattern":"import json, pathlib, re, subprocess\n\ntry:\n    apply_promoted_overlay(overlay, expected_target_bases=bases)\nexcept RuntimeError as exc:\n    msg = str(exc)\n    if \"rollback was incomplete\" in msg:\n        # the recovery JSON path is embedded after \"recovery:\"\n        m = re.search(r\"recovery:\\s*(.+?)(?:\\s*$)\", msg, re.M)\n        recovery_path = pathlib.Path(m.group(1).strip())\n        rec = json.loads(recovery_path.read_text())\n        # restore each destination from git as the source of truth\n        for r in rec[\"backups\"]:\n            subprocess.run([\"git\", \"-C\", str(repo_root), \"checkout\", \"HEAD\", \"--\", r[\"target\"]], check=True)\n        # then manually delete orphaned candidate/backup temp files listed in rec\n        raise SystemExit(f\"manual recovery required; see {recovery_path}\")\n    raise","preventionTips":["Never ignore this error — the working tree is in an indeterminate state until you restore from the recovery artifact.","Always restore destinations from `git` (the source of truth) rather than guessing which temp file is correct.","Fix the underlying cause (concurrent writer / unreliable FS) before retrying apply."],"tags":["integrity","rollback","recovery","transaction","manual-intervention"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}