Fleet manager for run

Error message

Fleet manager for run {} exited with open work; wait for stale reconciliation before resuming

What it means

The Fleet manager standby loop in drive/ensure-run ownership (crates/tui/src/fleet/manager.rs:767) refuses to take over a run whose previous manager process exited while tasks still had open (leased) work. A crash can leave orphan child processes alive; blindly relaunching their unchanged leased attempts would duplicate work. Stale reconciliation owns the recovery/generation transition, so the caller must wait for it before resuming.

Solutions

Let stale lease reconciliation run first (wait for lease deadlines to expire so the ledger reclaims/requeues the open work), then resume the run.
Check run_has_open_work / active leases before attempting to resume; if leases are fresh, wait instead of relaunching.
Kill any orphaned worker child processes from the crashed manager, then restart once the ledger shows no live leases.

Example fix

// before
let report = manager.ensure_run_driven(&run_id, ...)?; // bails after a crash with open leases
// after
while manager.run_has_open_work(&run_id)? {
    tokio::time::sleep(Duration::from_secs(5)).await; // let stale reconciliation reclaim leases
}
let report = manager.ensure_run_driven(&run_id, ...)?;

Defensive patterns

Strategy: retry

Validate before calling

if manager.run_has_open_work(&run_id)? {
    // leases from a previous manager are still live; wait for stale reconciliation first
}

Try / catch

loop {
    match ensure_run_driven(...).await {
        Ok(status) => break status,
        Err(e) if e.to_string().contains("exited with open work") => tokio::time::sleep(Duration::from_secs(5)).await,
        Err(e) => return Err(e.into()),
    }
}

Prevention

After a manager crash, always let stale lease deadlines expire (or run reconciliation) before resuming.
Clean up orphaned worker child processes before restarting a run.
Avoid two managers pointing at the same Fleet state directory; use one owner per run.

When it happens

Trigger: A second manager process acquires the file-manager lock after the first one exited (observed_owner == true) while ledger rebuild shows the run still has leased/in-flight tasks — i.e. resuming a run right after the owning manager crashed or was killed mid-lease.

Common situations: Restarting the TUI/CLI after a crash and immediately re-issuing the run command; two machines or shells pointing at the same Fleet state directory; an orphaned worker child keeping tasks leased after the parent died.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@73e0f67d83 (2026-09-22). Data as JSON: /api/errors/ca1fc69c67254f5d. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/src/fleet/manager.rs:767

                    .read(true)
                    .write(true)
                    .open(&path)
                    .with_context(|| format!("opening Fleet manager lock {}", path.display()))
            })
            .await
            .context("Fleet manager lock setup task failed to join")??
        };
        let mut manager_lock = fd_lock::RwLock::new(lock_file);
        let standby_interval = tick_interval
            .min(Duration::from_millis(100))
            .max(Duration::from_millis(10));
        let mut observed_owner = false;
        let _manager_guard = loop {
            match manager_lock.try_write() {
                Ok(guard) => {
                    if observed_owner {
                        if self.run_has_open_work(run_id)? {
                            bail!(
                                "Fleet manager for run {} exited with open work; wait for stale reconciliation before resuming",
                                run_id.0
                            );
                        }
                        return self.run_status(run_id);
                    }
                    break guard;
                }
                Err(err) if err.kind() == ErrorKind::WouldBlock => {
                    // Another process owns this run. Wait for it to finish,
                    // but never treat lock release as permission to relaunch
                    // its unchanged leased attempts: an orphan child may still
                    // be alive after a crash. Stale reconciliation owns that
                    // recovery/generation transition.
                    observed_owner = true;
                    if !self.run_has_open_work(run_id)? {
                        return self.run_status(run_id);
                    }

View on GitHub (pinned to 73e0f67d83)