{"record":{"id":"45e94b1b8633dd43","repo":"Hmbown/CodeWhale","slug":"fleet-worker-worker-id-coordination-state-is-bus","errorCode":null,"errorMessage":"Fleet worker {worker_id} coordination state is busy; retry restart","messagePattern":"Fleet worker (.+?) coordination state is busy; retry restart","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"crates/tui/src/fleet/manager.rs","lineNumber":1042,"sourceCode":"    pub fn restart_worker(&self, worker_id: &str) -> Result<FleetRestartReport> {\n        let state = self.ledger.rebuild_state()?;\n        let Some(task) = active_task_for_worker(&state, worker_id)\n            .or_else(|| latest_task_for_worker(&state, worker_id))\n        else {\n            bail!(\"worker {worker_id} has no fleet task to restart\");\n        };\n        let run = state\n            .runs\n            .get(&task.entry.run_id.0)\n            .ok_or_else(|| anyhow!(\"fleet run {} does not exist\", task.entry.run_id.0))?;\n        let max_workers = run\n            .max_workers\n            .unwrap_or_else(|| run.worker_specs.len().max(1))\n            .clamp(1, 128);\n        let mut coordination_guard = match &self.sub_agent_manager {\n            Some(manager) => {\n                let Ok(guard) = manager.try_write() else {\n                    bail!(\"Fleet worker {worker_id} coordination state is busy; retry restart\");\n                };\n                Some(guard)\n            }\n            None => None,\n        };\n        let now = timestamp();\n        let latest_seq = state\n            .latest_seq\n            .get(&event_key(\n                worker_id,\n                &task.entry.run_id.0,\n                &task.entry.task_id,\n            ))\n            .copied()\n            .unwrap_or(0);\n        let heartbeat_at = state\n            .heartbeats\n            .get(worker_id)","sourceCodeStart":1024,"sourceCodeEnd":1060,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/0c42157ee52f9d55af2b506d71b46249910f77d3/crates/tui/src/fleet/manager.rs#L1024-L1060","documentation":"restart_worker takes a non-blocking try_write on the sub-agent coordination manager (an RwLock) before mutating launch generations. If any other thread currently holds a read or write lock, the restart aborts immediately with 'coordination state is busy; retry restart' instead of blocking the operator path.","triggerScenarios":"Calling restart_worker while the sub-agent coordination manager is locked: a concurrent restart/launch, or the event-processing loop reading coordination records.","commonSituations":"UI issuing two restarts in quick succession; a scripted restart loop with no spacing; restart racing worker heartbeat processing.","solutions":["Retry restart_worker after a short backoff; the lock is short-lived by design","Serialize restarts per worker in calling code instead of issuing them concurrently","If it persists, check for a thread holding the coordination lock too long (a hung coordination record write)"],"exampleFix":"// before\nlet report = manager.restart_worker(&worker_id)?;\n\n// after\nlet mut backoff = Duration::from_millis(50);\nlet report = loop {\n    match manager.restart_worker(&worker_id) {\n        Ok(report) => break report,\n        Err(err) if err.to_string().contains(\"coordination state is busy\") => {\n            std::thread::sleep(backoff);\n            backoff = (backoff * 2).min(Duration::from_millis(800));\n        }\n        Err(err) => return Err(err),\n    }\n};","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"let mut backoff = std::time::Duration::from_millis(50);\nloop {\n    match manager.restart_worker(&worker_id) {\n        Ok(report) => break Ok(report),\n        Err(err) if err.to_string().contains(\"coordination state is busy\") => {\n            std::thread::sleep(backoff);\n            backoff = (backoff * 2).min(std::time::Duration::from_millis(800));\n        }\n        Err(err) => break Err(err),\n    }\n}","preventionTips":["Serialize restart requests per worker instead of firing them concurrently","Space scripted restarts by at least a few hundred milliseconds","Treat this message as transient by design; never surface it as a hard failure"],"tags":["fleet","lock-contention","retryable","rust","codewhale"],"backgroundTag":"resource-busy-retry","analyzedSha":"0c42157ee52f9d55af2b506d71b46249910f77d3","analyzedAt":"2026-08-20T21:50:45.477Z","schemaVersion":2},"datasetVersion":"2026-08-22T04:17:13.399Z"}