Fleet worker {worker_id} coordination state is busy; retry r

Error message

Fleet worker {worker_id} coordination state is busy; retry restart

What it means

restart_worker takes a non-blocking try_write on the sub-agent coordination manager (an RwLock) before mutating launch generations. If any other thread currently holds a read or write lock, the restart aborts immediately with 'coordination state is busy; retry restart' instead of blocking the operator path.

Source

Thrown at crates/tui/src/fleet/manager.rs:1042

    pub fn restart_worker(&self, worker_id: &str) -> Result<FleetRestartReport> {
        let state = self.ledger.rebuild_state()?;
        let Some(task) = active_task_for_worker(&state, worker_id)
            .or_else(|| latest_task_for_worker(&state, worker_id))
        else {
            bail!("worker {worker_id} has no fleet task to restart");
        };
        let run = state
            .runs
            .get(&task.entry.run_id.0)
            .ok_or_else(|| anyhow!("fleet run {} does not exist", task.entry.run_id.0))?;
        let max_workers = run
            .max_workers
            .unwrap_or_else(|| run.worker_specs.len().max(1))
            .clamp(1, 128);
        let mut coordination_guard = match &self.sub_agent_manager {
            Some(manager) => {
                let Ok(guard) = manager.try_write() else {
                    bail!("Fleet worker {worker_id} coordination state is busy; retry restart");
                };
                Some(guard)
            }
            None => None,
        };
        let now = timestamp();
        let latest_seq = state
            .latest_seq
            .get(&event_key(
                worker_id,
                &task.entry.run_id.0,
                &task.entry.task_id,
            ))
            .copied()
            .unwrap_or(0);
        let heartbeat_at = state
            .heartbeats
            .get(worker_id)

View on GitHub (pinned to 0c42157ee5)

Solutions

Retry restart_worker after a short backoff; the lock is short-lived by design
Serialize restarts per worker in calling code instead of issuing them concurrently
If it persists, check for a thread holding the coordination lock too long (a hung coordination record write)

Example fix

// before
let report = manager.restart_worker(&worker_id)?;

// after
let mut backoff = Duration::from_millis(50);
let report = loop {
    match manager.restart_worker(&worker_id) {
        Ok(report) => break report,
        Err(err) if err.to_string().contains("coordination state is busy") => {
            std::thread::sleep(backoff);
            backoff = (backoff * 2).min(Duration::from_millis(800));
        }
        Err(err) => return Err(err),
    }
};

Defensive patterns

Strategy: retry

Try / catch

let mut backoff = std::time::Duration::from_millis(50);
loop {
    match manager.restart_worker(&worker_id) {
        Ok(report) => break Ok(report),
        Err(err) if err.to_string().contains("coordination state is busy") => {
            std::thread::sleep(backoff);
            backoff = (backoff * 2).min(std::time::Duration::from_millis(800));
        }
        Err(err) => break Err(err),
    }
}

Prevention

Serialize restart requests per worker instead of firing them concurrently
Space scripted restarts by at least a few hundred milliseconds
Treat this message as transient by design; never surface it as a hard failure

When it happens

Trigger: Calling restart_worker while the sub-agent coordination manager is locked: a concurrent restart/launch, or the event-processing loop reading coordination records.

Common situations: UI issuing two restarts in quick succession; a scripted restart loop with no spacing; restart racing worker heartbeat processing.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/45e94b1b8633dd43. Report an issue: GitHub.