spacedriveapp/spacedrive · critical

Worker channel closed trying to steal task

Error message

Worker channel closed trying to steal task

What it means

WorkerComm::steal_task sends a StealRequest to another worker's command channel and panics via .expect() if that channel is closed. The problem is structural: every worker's WorkStealer holds an Arc<Vec<WorkerComm>> built once at system creation and never pruned (worker/mod.rs:241-258), so once any worker's channel closes, every idle worker's periodic steal sweep (idle_check every second, run.rs:124 -> runner.rs:906-929 -> WorkStealer::steal) eventually reaches the dead comm and the send at mod.rs:227-234 panics.

Source

Thrown at crates/task-system/src/worker/mod.rs:234

	msgs_tx: chan::Sender<WorkerMessage<E>>,
}

impl<E: RunError> WorkerComm<E> {
	pub async fn steal_task(
		&self,
		stealer_id: WorkerId,
		stolen_task_tx: chan::Sender<Option<StoleTaskMessage<E>>>,
	) -> bool {
		let (tx, rx) = oneshot::channel();

		self.msgs_tx
			.send(WorkerMessage::StealRequest {
				stealer_id,
				ack: tx,
				stolen_task_tx,
			})
			.await
			.expect("Worker channel closed trying to steal task");

		rx.await
			.expect("Worker channel closed trying to steal task")
	}
}

pub struct WorkStealer<E: RunError> {
	worker_comms: Arc<Vec<WorkerComm<E>>>,
}

impl<E: RunError> Clone for WorkStealer<E> {
	fn clone(&self) -> Self {
		Self {
			worker_comms: Arc::clone(&self.worker_comms),
		}
	}
}

View on GitHub (pinned to 6dfeccf211)

Solutions

1. Library-level fix: skip dead comms in the sweep - check worker_comm.msgs_tx.is_closed() before steal_task and continue to the next worker
2. Library-level fix: replace .expect with returning false (treat an unreachable worker as 'nothing to steal'), matching the existing graceful handling at runner.rs:778-798
3. Library-level fix: have SystemHandle::shutdown set has_shutdown (it does, system.rs:218) and make idle_check stop dispatching steals once set, or abort all current_steal_task_handle's before joining workers
4. As an operator: ensure workers are only shut down together via system.shutdown(), never individually, and keep the pool topology fixed for the process lifetime

Example fix

// before (worker/mod.rs:227-234)
self.msgs_tx
    .send(WorkerMessage::StealRequest { stealer_id, ack: tx, stolen_task_tx })
    .await
    .expect("Worker channel closed trying to steal task");

// after
if self
    .msgs_tx
    .send(WorkerMessage::StealRequest { stealer_id, ack: tx, stolen_task_tx })
    .await
    .is_err()
{
    trace!(worker_id = self.worker_id, "Target worker channel closed; treating as no task stolen");
    return false;
}

// Not fully preventable from outside: WorkStealer's Arc<Vec<WorkerComm>> is
// library-internal and never pruned, so you cannot check is_closed() yourself.
// The one thing you CAN validate: only ever shut the pool down as a whole.

// safe: coordinated, single entry point
// system.handle().shutdown().await;

// unsafe (leaves dead comms in live stealers): shutting workers individually

// The panic fires inside a detached library-spawned task (runner.rs:1402-1407),
// so catch_unwind at the call site cannot intercept it. Contain and detect via
// a panic hook instead:
std::panic::set_hook(Box::new(|info| {
    let msg = info.to_string();
    if msg.contains("Worker channel closed trying to steal task") {
        tracing::warn!("steal sweep hit a dead worker; stealing degraded - expect library fix");
        return;
    }
    eprintln!("panic: {msg}");
}));

When it happens

Trigger: Partial pool shutdown while other workers still run and steal: SystemHandle::shutdown shuts all workers down concurrently (join_all, system.rs:227), so a still-alive worker's idle steal sweep can target a worker that already finished shutting down. The panic kills the detached steal task spawned at runner.rs:1402-1407, and because its JoinHandle stays in current_steal_task_handle and idle_check only re-steals when that handle is None (runner.rs:906-910), that worker silently stops stealing forever.

Common situations: Multi-worker teardown races during system shutdown; embedding scenarios where a worker task dies (panic-restart or runtime drop) while siblings continue; any future feature that shuts down individual workers. Symptom is subtle: no crash of the main loop, but work distribution stops for affected workers.

Related errors

• Worker channel closed trying to pause a not running task
• Worker channel closed trying to cancel a not running task
• Worker channel closed trying to force task abortion
• Worker channel closed trying to shutdown
• Stolen task channel closed

AI-assisted analysis of spacedriveapp/spacedrive@6dfeccf211 (2026-08-16). Data as JSON: /api/errors/cd2fb0ab3d32f4db. Report an issue: GitHub.