worker {worker_id} task changed before it could be restarted

Error message

worker {worker_id} task changed before it could be restarted

What it means

restart_worker performs a compare-and-swap style update guarded by ledger sequence numbers; the closure returned restarted=false, meaning the task's lease, attempt, or generation changed between the state snapshot and the committed update. Someone else (worker events, another operator action) mutated the task first.

Source

Thrown at crates/tui/src/fleet/manager.rs:1090

            task.entry.attempts,
            || {
                if let Some(guard) = coordination_guard.as_mut() {
                    let task_spec = run
                        .task_specs
                        .iter()
                        .find(|spec| spec.id == task.entry.task_id)
                        .ok_or_else(|| {
                            anyhow!("fleet task {} does not exist", task.entry.task_id)
                        })?;
                    self.prepare_registered_restart_generation(
                        guard, &state, task, task_spec, worker_id,
                    )?;
                }
                Ok(())
            },
        )?;
        if !restarted {
            bail!("worker {worker_id} task changed before it could be restarted");
        }
        self.ledger
            .update_run_status(&task.entry.run_id, FleetRunStatus::Running, &timestamp())?;
        Ok(FleetRestartReport {
            run_id: task.entry.run_id.clone(),
            max_workers,
            inspection: self.inspect_worker(worker_id)?,
        })
    }

    /// Prepare or consume the exact durable launch generation for one Fleet
    /// retry. The persisted one-generation-ahead record is the prepare marker:
    /// it is not launchable while the ledger remains on the old attempt, but a
    /// retry after a crash may validate and consume it idempotently.
    fn prepare_registered_restart_generation(
        &self,
        coordination: &mut SubAgentManager,
        state: &FleetLedgerState,

View on GitHub (pinned to 0c42157ee5)

Solutions

Re-inspect the worker to learn its new task/attempt state, then retry restart_worker if still appropriate
Avoid issuing overlapping control commands for the same worker
If this fires constantly, look for a writer loop that keeps bumping the task generation

Defensive patterns

Strategy: retry

Try / catch

match manager.restart_worker(&worker_id) {
    Ok(report) => Ok(report),
    Err(err) if err.to_string().contains("task changed before it could be restarted") => {
        let inspection = manager.inspect_worker(&worker_id)?; // see what it is now
        Ok(/* decide based on inspection */)
    }
    Err(err) => Err(err),
}

Prevention

Do not stack restart, interrupt, and stop commands on one worker simultaneously
Re-inspect after any control-command race instead of assuming the old task

When it happens

Trigger: Calling restart_worker concurrently with task completion, lease expiry/reassignment, or another restart/interrupt on the same worker.

Common situations: Restart racing the worker's own completion event; two UI panes restarting the same worker; restart issued just as stale reconciliation reaps the lease.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/973d48d0ff7f546a. Report an issue: GitHub.