Hmbown/CodeWhale · error

fleet run {} is already terminal ({lifecycle:?})

Error message

fleet run {} is already terminal ({lifecycle:?})

What it means

FleetManager::activate_run refuses to activate a run whose effective lifecycle is Completed, Failed, or Cancelled. The lifecycle comes from run_status_overrides if present, otherwise the derived run.status. Terminal runs are never reactivated because their ledger is closed; new work requires a new run or the explicit worker-restart control path.

Source

Thrown at crates/tui/src/fleet/manager.rs:467

    pub fn activate_run(&self, run_id: &FleetRunId) -> Result<FleetRunReport> {
        let state = self.ledger.rebuild_state()?;
        let run =
            state
                .runs
                .get(&run_id.0)
                .cloned()
                .ok_or_else(|| FleetControlError::UnknownRun {
                    run_id: run_id.0.clone(),
                })?;
        let lifecycle = state
            .run_status_overrides
            .get(&run_id.0)
            .unwrap_or(&run.status);
        if matches!(
            lifecycle,
            FleetRunStatus::Completed | FleetRunStatus::Failed | FleetRunStatus::Cancelled
        ) {
            bail!("fleet run {} is already terminal ({lifecycle:?})", run_id.0);
        }
        if !matches!(lifecycle, FleetRunStatus::Running) {
            self.ledger
                .update_run_status(run_id, FleetRunStatus::Running, &timestamp())?;
        }
        let state = self.ledger.rebuild_state()?;
        let snapshot = self.status_from_state(Some(run_id), &state);
        Ok(FleetRunReport {
            run_id: run.id,
            task_count: run.task_specs.len(),
            leased: 0,
            queued: snapshot.queued,
            worker_ids: run
                .worker_specs
                .iter()
                .map(|worker| worker.id.clone())
                .collect(),
            warnings: Vec::new(),

View on GitHub (pinned to 0c42157ee5)

Solutions

1. Check run status first (status_from_state / run_status) and only call activate_run for non-terminal runs
2. Create a new fleet run with the same task specs instead of re-activating a terminal one
3. If the run was cancelled by mistake and must continue, use the documented worker restart path rather than activation
4. If you believe the terminal override is stale, reconcile the ledger state before retrying

Example fix

// before
let report = manager.activate_run(&run_id)?; // panics path: already terminal

// after
let status = manager.run_status(&run_id)?;
if !matches!(status.lifecycle, FleetRunStatus::Completed | FleetRunStatus::Failed | FleetRunStatus::Cancelled) {
    let report = manager.activate_run(&run_id)?;
} else {
    // start a fresh run with the same specs
}

fn run_is_terminal(status: &FleetRunStatus) -> bool {
    matches!(
        status,
        FleetRunStatus::Completed | FleetRunStatus::Failed | FleetRunStatus::Cancelled
    )
}

// before activating:
let st = manager.run_status(&run_id)?;
assert!(!run_is_terminal(&st.lifecycle));

match manager.activate_run(&run_id) {
    Ok(report) => { /* activated */ }
    Err(err) if err.to_string().contains("already terminal") => {
        // create a new run with the same specs instead
    }
    Err(err) => return Err(err),
}

When it happens

Trigger: Calling FleetManager::activate_run(&run_id) on a run that finished (Completed/Failed) or was cancelled via stop_run/stop_all, including cases where a status override marks it terminal while the derived status does not.

Common situations: Retrying a failed fleet run by re-activating the same run id; a cancelled run whose id was reused from shell history; UI re-sending an activate command after the run completed in the background.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• fleet manager for run {} exited with open work; wait for sta
• worker {worker_id} no longer has that running fleet task
• worker {worker_id} has no fleet task to restart
• Fleet worker {worker_id} coordination state is busy; retry r
• worker {worker_id} task changed before it could be restarted

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/36c81a6611060d5e. Report an issue: GitHub.