abhigyanpatwari/GitNexus · critical · RuntimeError

overlay apply failed and rollback was incomplete; recovery:

Error message

overlay apply failed and rollback was incomplete; recovery: {recovery}

What it means

Thrown from the `except` block of `apply_promoted_overlay` when the apply failed and one or more completed replacements could not be cleanly rolled back (`rollback_failures` non-empty). A recovery JSON artifact listing orphaned candidate/backup files is written to the repo root and its path is included in the message — manual intervention is required because the working tree is in an indeterminate state.

Source

Thrown at eval/workflow_bench/promotion_apply.py:874

                    replacement["name"],
                )
            except BaseException as rollback_exc:
                if not rollback_exchange_is_valid(replacement, temporary_identity, displaced_state):
                    rollback_failures.append(f"{replacement['target']}: {type(rollback_exc).__name__}: {rollback_exc}")
            else:
                if not rollback_exchange_is_valid(replacement, temporary_identity, displaced_state):
                    rollback_failures.append(f"{replacement['target']}: rollback parity check failed")
        if rollback_failures:
            preserve_backups = True
            recovery = _write_recovery_artifact(
                root_descriptor,
                repo_root,
                failure=exc,
                rollback_failures=rollback_failures,
                replacements=replacements,
                transaction_state="rollback-incomplete",
            )
            raise RuntimeError(f"overlay apply failed and rollback was incomplete; recovery: {recovery}") from exc
        rollback_complete = True
        if isinstance(exc, (KeyboardInterrupt, SystemExit)):
            raise
        raise RuntimeError("overlay apply failed and all replacements were rolled back") from exc
    finally:
        active_failure = sys.exc_info()[1]
        try:
            if not preserve_backups:
                cleanup_failures: list[str] = []
                cleanup_exception: BaseException | None = None
                for replacement in replacements:
                    for temporary in (replacement["candidate"], replacement["backup"]):
                        if temporary is None:
                            continue
                        try:
                            _unlink_temporary(replacement["parent_descriptor"], temporary)
                        except BaseException as cleanup_exc:
                            cleanup_exception = cleanup_exc

View on GitHub (pinned to d540b00184)

Solutions

1. Read the recovery JSON named in the message — it lists each target with `candidate_exists`/`backup_exists` so you can see which slots have orphaned temp files.
2. Inspect each named destination and decide manually whether the candidate or the backup represents the correct state; restore from `git` if unsure.
3. Delete the orphaned `.wfbench-overlay-recovery-*` candidate/backup files only after restoring destinations.
4. Fix the root cause (serialize writers / move to a reliable FS — see 411) before retrying apply.

Example fix

# 1. open the recovery file named in the error
import json, pathlib
rec = json.loads(pathlib.Path(recovery_path_in_message).read_text())
# 2. for each record, restore the destination from git as the source of truth
import subprocess
for r in rec['backups']:
    subprocess.run(['git', 'checkout', 'HEAD', '--', r['target']], check=True)
# 3. remove orphaned candidate/backup temp files, then delete the recovery json

null

null

import json, pathlib, re, subprocess

try:
    apply_promoted_overlay(overlay, expected_target_bases=bases)
except RuntimeError as exc:
    msg = str(exc)
    if "rollback was incomplete" in msg:
        # the recovery JSON path is embedded after "recovery:"
        m = re.search(r"recovery:\s*(.+?)(?:\s*$)", msg, re.M)
        recovery_path = pathlib.Path(m.group(1).strip())
        rec = json.loads(recovery_path.read_text())
        # restore each destination from git as the source of truth
        for r in rec["backups"]:
            subprocess.run(["git", "-C", str(repo_root), "checkout", "HEAD", "--", r["target"]], check=True)
        # then manually delete orphaned candidate/backup temp files listed in rec
        raise SystemExit(f"manual recovery required; see {recovery_path}")
    raise

When it happens

Trigger: Any failure (drift, parity, exception) during the apply loop after at least one exchange completed, where the rollback pass found that re-swapping a target did not restore the expected displaced state (see `rollback_exchange_is_valid` returning False at :683-687). The transaction is left 'rollback-incomplete' and a `.wfbench-overlay-recovery-*.json` is written.

Common situations: Filesystem that does not honor RENAME_EXCHANGE during rollback; concurrent writer corrupted a slot mid-rollback; storage failure; the original swap landed but the rollback swap raced with another edit. Whatever the cause, do NOT assume the tree is clean.

Related errors

• overlay apply failed and all replacements were rolled back
• repository root changed during overlay {phase}: {root}
• overlay destination parent changed during {phase}: {item['ta
• overlay destination drifted during apply: {replacement['targ
• post-apply parity check failed: {replacement['target']}

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/4dab9ecd76cc6ad1. Report an issue: GitHub.