{"record":{"id":"cd2fb0ab3d32f4db","repo":"spacedriveapp/spacedrive","slug":"worker-channel-closed-trying-to-steal-task","errorCode":null,"errorMessage":"Worker channel closed trying to steal task","messagePattern":"Worker channel closed trying to steal task","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/task-system/src/worker/mod.rs","lineNumber":234,"sourceCode":"\tmsgs_tx: chan::Sender<WorkerMessage<E>>,\n}\n\nimpl<E: RunError> WorkerComm<E> {\n\tpub async fn steal_task(\n\t\t&self,\n\t\tstealer_id: WorkerId,\n\t\tstolen_task_tx: chan::Sender<Option<StoleTaskMessage<E>>>,\n\t) -> bool {\n\t\tlet (tx, rx) = oneshot::channel();\n\n\t\tself.msgs_tx\n\t\t\t.send(WorkerMessage::StealRequest {\n\t\t\t\tstealer_id,\n\t\t\t\tack: tx,\n\t\t\t\tstolen_task_tx,\n\t\t\t})\n\t\t\t.await\n\t\t\t.expect(\"Worker channel closed trying to steal task\");\n\n\t\trx.await\n\t\t\t.expect(\"Worker channel closed trying to steal task\")\n\t}\n}\n\npub struct WorkStealer<E: RunError> {\n\tworker_comms: Arc<Vec<WorkerComm<E>>>,\n}\n\nimpl<E: RunError> Clone for WorkStealer<E> {\n\tfn clone(&self) -> Self {\n\t\tSelf {\n\t\t\tworker_comms: Arc::clone(&self.worker_comms),\n\t\t}\n\t}\n}\n","sourceCodeStart":216,"sourceCodeEnd":252,"githubUrl":"https://github.com/spacedriveapp/spacedrive/blob/6dfeccf2113039e35f2ce735f945e70dc3e4ea45/crates/task-system/src/worker/mod.rs#L216-L252","documentation":"WorkerComm::steal_task sends a StealRequest to another worker's command channel and panics via .expect() if that channel is closed. The problem is structural: every worker's WorkStealer holds an Arc<Vec<WorkerComm>> built once at system creation and never pruned (worker/mod.rs:241-258), so once any worker's channel closes, every idle worker's periodic steal sweep (idle_check every second, run.rs:124 -> runner.rs:906-929 -> WorkStealer::steal) eventually reaches the dead comm and the send at mod.rs:227-234 panics.","triggerScenarios":"Partial pool shutdown while other workers still run and steal: SystemHandle::shutdown shuts all workers down concurrently (join_all, system.rs:227), so a still-alive worker's idle steal sweep can target a worker that already finished shutting down. The panic kills the detached steal task spawned at runner.rs:1402-1407, and because its JoinHandle stays in current_steal_task_handle and idle_check only re-steals when that handle is None (runner.rs:906-910), that worker silently stops stealing forever.","commonSituations":"Multi-worker teardown races during system shutdown; embedding scenarios where a worker task dies (panic-restart or runtime drop) while siblings continue; any future feature that shuts down individual workers. Symptom is subtle: no crash of the main loop, but work distribution stops for affected workers.","solutions":["Library-level fix: skip dead comms in the sweep - check worker_comm.msgs_tx.is_closed() before steal_task and continue to the next worker","Library-level fix: replace .expect with returning false (treat an unreachable worker as 'nothing to steal'), matching the existing graceful handling at runner.rs:778-798","Library-level fix: have SystemHandle::shutdown set has_shutdown (it does, system.rs:218) and make idle_check stop dispatching steals once set, or abort all current_steal_task_handle's before joining workers","As an operator: ensure workers are only shut down together via system.shutdown(), never individually, and keep the pool topology fixed for the process lifetime"],"exampleFix":"// before (worker/mod.rs:227-234)\nself.msgs_tx\n    .send(WorkerMessage::StealRequest { stealer_id, ack: tx, stolen_task_tx })\n    .await\n    .expect(\"Worker channel closed trying to steal task\");\n\n// after\nif self\n    .msgs_tx\n    .send(WorkerMessage::StealRequest { stealer_id, ack: tx, stolen_task_tx })\n    .await\n    .is_err()\n{\n    trace!(worker_id = self.worker_id, \"Target worker channel closed; treating as no task stolen\");\n    return false;\n}","handlingStrategy":"fallback","validationCode":"// Not fully preventable from outside: WorkStealer's Arc<Vec<WorkerComm>> is\n// library-internal and never pruned, so you cannot check is_closed() yourself.\n// The one thing you CAN validate: only ever shut the pool down as a whole.\n\n// safe: coordinated, single entry point\n// system.handle().shutdown().await;\n\n// unsafe (leaves dead comms in live stealers): shutting workers individually","typeGuard":null,"tryCatchPattern":"// The panic fires inside a detached library-spawned task (runner.rs:1402-1407),\n// so catch_unwind at the call site cannot intercept it. Contain and detect via\n// a panic hook instead:\nstd::panic::set_hook(Box::new(|info| {\n    let msg = info.to_string();\n    if msg.contains(\"Worker channel closed trying to steal task\") {\n        tracing::warn!(\"steal sweep hit a dead worker; stealing degraded - expect library fix\");\n        return;\n    }\n    eprintln!(\"panic: {msg}\");\n}));","preventionTips":["Never shut down workers individually; use only SystemHandle::shutdown so the whole pool drains together","Keep the worker pool topology fixed for the process lifetime; do not add per-worker lifecycle management on top of this crate","Watch for silent symptom: a worker that stops stealing after this panic (its steal handle is never cleared) - monitor throughput per worker","Patch upstream: skip is_closed() comms in the sweep and return false on SendError instead of expecting"],"tags":["rust","tokio","async","panic","channel","work-stealing","shutdown","race-condition","task-system"],"backgroundTag":null,"analyzedSha":"6dfeccf2113039e35f2ce735f945e70dc3e4ea45","analyzedAt":"2026-08-16T11:26:17.074Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}