{"record":{"id":"ca1fc69c67254f5d","repo":"Hmbown/CodeWhale","slug":"fleet-manager-for-run-exited-with-open-work-wait-for-stale","errorCode":null,"errorMessage":"Fleet manager for run {} exited with open work; wait for stale reconciliation before resuming","messagePattern":"Fleet manager for run (.+?) exited with open work; wait for stale reconciliation before resuming","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/tui/src/fleet/manager.rs","lineNumber":767,"sourceCode":"                    .read(true)\n                    .write(true)\n                    .open(&path)\n                    .with_context(|| format!(\"opening Fleet manager lock {}\", path.display()))\n            })\n            .await\n            .context(\"Fleet manager lock setup task failed to join\")??\n        };\n        let mut manager_lock = fd_lock::RwLock::new(lock_file);\n        let standby_interval = tick_interval\n            .min(Duration::from_millis(100))\n            .max(Duration::from_millis(10));\n        let mut observed_owner = false;\n        let _manager_guard = loop {\n            match manager_lock.try_write() {\n                Ok(guard) => {\n                    if observed_owner {\n                        if self.run_has_open_work(run_id)? {\n                            bail!(\n                                \"Fleet manager for run {} exited with open work; wait for stale reconciliation before resuming\",\n                                run_id.0\n                            );\n                        }\n                        return self.run_status(run_id);\n                    }\n                    break guard;\n                }\n                Err(err) if err.kind() == ErrorKind::WouldBlock => {\n                    // Another process owns this run. Wait for it to finish,\n                    // but never treat lock release as permission to relaunch\n                    // its unchanged leased attempts: an orphan child may still\n                    // be alive after a crash. Stale reconciliation owns that\n                    // recovery/generation transition.\n                    observed_owner = true;\n                    if !self.run_has_open_work(run_id)? {\n                        return self.run_status(run_id);\n                    }","sourceCodeStart":749,"sourceCodeEnd":785,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/src/fleet/manager.rs#L749-L785","documentation":"The Fleet manager standby loop in drive/ensure-run ownership (crates/tui/src/fleet/manager.rs:767) refuses to take over a run whose previous manager process exited while tasks still had open (leased) work. A crash can leave orphan child processes alive; blindly relaunching their unchanged leased attempts would duplicate work. Stale reconciliation owns the recovery/generation transition, so the caller must wait for it before resuming.","triggerScenarios":"A second manager process acquires the file-manager lock after the first one exited (observed_owner == true) while ledger rebuild shows the run still has leased/in-flight tasks — i.e. resuming a run right after the owning manager crashed or was killed mid-lease.","commonSituations":"Restarting the TUI/CLI after a crash and immediately re-issuing the run command; two machines or shells pointing at the same Fleet state directory; an orphaned worker child keeping tasks leased after the parent died.","solutions":["Let stale lease reconciliation run first (wait for lease deadlines to expire so the ledger reclaims/requeues the open work), then resume the run.","Check run_has_open_work / active leases before attempting to resume; if leases are fresh, wait instead of relaunching.","Kill any orphaned worker child processes from the crashed manager, then restart once the ledger shows no live leases."],"exampleFix":"// before\nlet report = manager.ensure_run_driven(&run_id, ...)?; // bails after a crash with open leases\n// after\nwhile manager.run_has_open_work(&run_id)? {\n    tokio::time::sleep(Duration::from_secs(5)).await; // let stale reconciliation reclaim leases\n}\nlet report = manager.ensure_run_driven(&run_id, ...)?;","handlingStrategy":"retry","validationCode":"if manager.run_has_open_work(&run_id)? {\n    // leases from a previous manager are still live; wait for stale reconciliation first\n}","typeGuard":null,"tryCatchPattern":"loop {\n    match ensure_run_driven(...).await {\n        Ok(status) => break status,\n        Err(e) if e.to_string().contains(\"exited with open work\") => tokio::time::sleep(Duration::from_secs(5)).await,\n        Err(e) => return Err(e.into()),\n    }\n}","preventionTips":["After a manager crash, always let stale lease deadlines expire (or run reconciliation) before resuming.","Clean up orphaned worker child processes before restarting a run.","Avoid two managers pointing at the same Fleet state directory; use one owner per run."],"tags":["fleet","concurrency","crash-recovery"],"backgroundTag":"invalid-state-transition","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-22T06:17:15.046Z"}