{"record":{"id":"870dd4e5a5c390aa","repo":"Hmbown/CodeWhale","slug":"fleet-manager-for-run-exited-with-open-work-wa","errorCode":null,"errorMessage":"fleet manager for run {} exited with open work; wait for stale reconciliation before resuming","messagePattern":"fleet manager for run (.+?) exited with open work; wait for stale reconciliation before resuming","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/tui/src/fleet/manager.rs","lineNumber":741,"sourceCode":"            .create(true)\n            .truncate(false)\n            .read(true)\n            .write(true)\n            .open(&manager_lock_path)\n            .with_context(|| {\n                format!(\"opening fleet manager lock {}\", manager_lock_path.display())\n            })?;\n        let mut manager_lock = fd_lock::RwLock::new(lock_file);\n        let standby_interval = tick_interval\n            .min(Duration::from_millis(100))\n            .max(Duration::from_millis(10));\n        let mut observed_owner = false;\n        let _manager_guard = loop {\n            match manager_lock.try_write() {\n                Ok(guard) => {\n                    if observed_owner {\n                        if self.run_has_open_work(run_id)? {\n                            bail!(\n                                \"fleet manager for run {} exited with open work; wait for stale reconciliation before resuming\",\n                                run_id.0\n                            );\n                        }\n                        return self.run_status(run_id);\n                    }\n                    break guard;\n                }\n                Err(err) if err.kind() == ErrorKind::WouldBlock => {\n                    // Another process owns this run. Wait for it to finish,\n                    // but never treat lock release as permission to relaunch\n                    // its unchanged leased attempts: an orphan child may still\n                    // be alive after a crash. Stale reconciliation owns that\n                    // recovery/generation transition.\n                    observed_owner = true;\n                    if !self.run_has_open_work(run_id)? {\n                        return self.run_status(run_id);\n                    }","sourceCodeStart":723,"sourceCodeEnd":759,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/0c42157ee52f9d55af2b506d71b46249910f77d3/crates/tui/src/fleet/manager.rs#L723-L759","documentation":"While attaching to a run, the manager loop saw another process holding the run's fd-lock (observed_owner), then acquired it after release, but run_has_open_work still reports leased/unreconciled tasks. Because lock release is not proof the previous owner's children are gone, Codewhale refuses to resume until stale lease reconciliation has reaped the orphaned work.","triggerScenarios":"Resuming or attaching to a fleet run right after the previous Codewhale process crashed, was killed, or exited while workers held leased tasks; the file lock was released but ledger leases have not timed out yet.","commonSituations":"Killing the TUI mid-run and immediately re-opening the same run; a machine sleep/shutdown that orphaned SSH fleet workers; two Codewhale instances racing over the same run directory.","solutions":["Wait for the stale-lease reconciliation window to elapse (leases expire by age), then retry the attach/resume","Confirm no orphan codewhale worker processes remain (ps / ssh host ps) and stop them","Verify you are not running two managers against the same run directory concurrently","If workers are provably dead and leases still will not reconcile, inspect the ledger events for stuck lease records"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match manager.resume_run(&run_id) {\n    Ok(report) => Ok(report),\n    Err(err) if err.to_string().contains(\"wait for stale reconciliation\") => {\n        // schedule a retry after the stale-lease window elapses\n        Err(err)\n    }\n    Err(err) => Err(err),\n}","preventionTips":["Shut down fleet runs cleanly (stop_run) before killing the process","Never run two Codewhale managers against one run directory","After a crash, wait out the lease timeout before resuming instead of retrying immediately"],"tags":["fleet","distributed-lock","stale-state","crash-recovery","rust","codewhale"],"backgroundTag":"stale-lock-detected","analyzedSha":"0c42157ee52f9d55af2b506d71b46249910f77d3","analyzedAt":"2026-08-20T21:50:45.477Z","schemaVersion":2},"datasetVersion":"2026-08-21T23:17:16.201Z"}