fleet manager for run

Error message

fleet manager for run {} exited with open work; wait for stale reconciliation before resuming

What it means

While attaching to a run, the manager loop saw another process holding the run's fd-lock (observed_owner), then acquired it after release, but run_has_open_work still reports leased/unreconciled tasks. Because lock release is not proof the previous owner's children are gone, Codewhale refuses to resume until stale lease reconciliation has reaped the orphaned work.

Solutions

Wait for the stale-lease reconciliation window to elapse (leases expire by age), then retry the attach/resume
Confirm no orphan codewhale worker processes remain (ps / ssh host ps) and stop them
Verify you are not running two managers against the same run directory concurrently
If workers are provably dead and leases still will not reconcile, inspect the ledger events for stuck lease records

Defensive patterns

Strategy: retry

Try / catch

match manager.resume_run(&run_id) {
    Ok(report) => Ok(report),
    Err(err) if err.to_string().contains("wait for stale reconciliation") => {
        // schedule a retry after the stale-lease window elapses
        Err(err)
    }
    Err(err) => Err(err),
}

Prevention

Shut down fleet runs cleanly (stop_run) before killing the process
Never run two Codewhale managers against one run directory
After a crash, wait out the lease timeout before resuming instead of retrying immediately

When it happens

Trigger: Resuming or attaching to a fleet run right after the previous Codewhale process crashed, was killed, or exited while workers held leased tasks; the file lock was released but ledger leases have not timed out yet.

Common situations: Killing the TUI mid-run and immediately re-opening the same run; a machine sleep/shutdown that orphaned SSH fleet workers; two Codewhale instances racing over the same run directory.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/870dd4e5a5c390aa. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/src/fleet/manager.rs:741

            .create(true)
            .truncate(false)
            .read(true)
            .write(true)
            .open(&manager_lock_path)
            .with_context(|| {
                format!("opening fleet manager lock {}", manager_lock_path.display())
            })?;
        let mut manager_lock = fd_lock::RwLock::new(lock_file);
        let standby_interval = tick_interval
            .min(Duration::from_millis(100))
            .max(Duration::from_millis(10));
        let mut observed_owner = false;
        let _manager_guard = loop {
            match manager_lock.try_write() {
                Ok(guard) => {
                    if observed_owner {
                        if self.run_has_open_work(run_id)? {
                            bail!(
                                "fleet manager for run {} exited with open work; wait for stale reconciliation before resuming",
                                run_id.0
                            );
                        }
                        return self.run_status(run_id);
                    }
                    break guard;
                }
                Err(err) if err.kind() == ErrorKind::WouldBlock => {
                    // Another process owns this run. Wait for it to finish,
                    // but never treat lock release as permission to relaunch
                    // its unchanged leased attempts: an orphan child may still
                    // be alive after a crash. Stale reconciliation owns that
                    // recovery/generation transition.
                    observed_owner = true;
                    if !self.run_has_open_work(run_id)? {
                        return self.run_status(run_id);
                    }

View on GitHub (pinned to 0c42157ee5)