{"record":{"id":"8d179798c89b6012","repo":"nautechsystems/nautilus_trader","slug":"old-user-stream-dispatch-task-did-not-stop-after-a","errorCode":null,"errorMessage":"old user stream dispatch task did not stop after abort","messagePattern":"old user stream dispatch task did not stop after abort","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/binance/src/futures/websocket/streams/recovery.rs","lineNumber":255,"sourceCode":"            .context(\"failed to close old user data WebSocket\")?;\n    }\n\n    // Drain queued events from the old stream while the replacement buffers new events.\n    let mut task_slot = ctx.ws_task.lock().await;\n    if let Some(outcome) = finish_task(\n        &mut task_slot,\n        Duration::from_secs(2),\n        Duration::from_secs(2),\n    )\n    .await\n    {\n        match outcome {\n            TaskJoinOutcome::Completed(()) | TaskJoinOutcome::Aborted => {}\n            TaskJoinOutcome::Failed(error) => {\n                anyhow::bail!(\"old user stream dispatch task failed: {error}\");\n            }\n            TaskJoinOutcome::Incomplete => {\n                anyhow::bail!(\"old user stream dispatch task did not stop after abort\");\n            }\n        }\n    }\n\n    let mut new_task = TaskSlot::new();\n    new_task\n        .spawn(run_user_stream_dispatch(\n            new_stream,\n            ctx.dispatch_ctx.clone(),\n            ctx.recovery_tx.clone(),\n            dispatch_fn,\n        ))\n        .map_err(|e| anyhow::anyhow!(\"failed to start recovered user stream dispatch task: {e}\"))?;\n\n    *ctx.ws_client.lock() = Some(new_ws);\n    *task_slot = new_task;\n    *ctx.listen_key.write() = Some(new_listen_key);\n    *ctx.recovery_listen_key.write() = None;","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/binance/src/futures/websocket/streams/recovery.rs#L237-L273","documentation":"During Binance Futures listen-key recovery, the old user-data dispatch task is aborted and given 2 seconds to be aborted and 2 seconds to be joined. If it neither completes nor aborts within those windows, finish_task reports TaskJoinOutcome::Incomplete and recovery bails so it can be retried with backoff, instead of spawning a second concurrent dispatcher on top of a stuck one.","triggerScenarios":"recovery is triggered (keepalive failure/expiry); the old dispatch task ignores the abort signal and stays alive past both finish_task timeouts — typically because it is blocked in a non-cancellation-aware await (e.g. a blocking send or a lock held by another holder).","commonSituations":"A downstream channel consumer is gone so the dispatcher blocks on an unbounded-channel send or receiver that never yields; a lock in DispatchCtx is held across a slow await; extremely slow/event-loop-starved runtime under heavy load.","solutions":["Check what the old dispatch task is blocked on — usually a channel send or a lock in DispatchCtx — and make it cancellation-aware (tokio::select! on cancellation_token)","Ensure no other task holds the DispatchCtx/shared locks for long periods so the dispatcher can observe abort","Increase tolerance by fixing runtime starvation: avoid blocking calls (std::thread::sleep, heavy CPU) on the tokio workers","If persistent, restart the node and report the stuck-task pattern to maintainers with logs"],"exampleFix":"// before: dispatcher blocks on send without observing cancellation\n_tx.send(event).await;\n// after: select on the abort signal\nlet _ = tokio::select! { _ = cancel.cancelled() => return, res = _tx.send(event) => res };","handlingStrategy":"retry","validationCode":"// Users cannot pre-validate the internal task lifecycle; ensure no custom code\n// holds DispatchCtx locks across awaits and avoid blocking calls on the runtime:\n// e.g. never call std::thread::sleep inside handlers; use tokio::time::sleep","typeGuard":null,"tryCatchPattern":"// Recovery retries automatically; on the ops side, alert if the same\n// \"did not stop after abort\" message repeats across attempts:\n// grep 'did not stop after abort' logs | wc -l  # >1 => investigate stuck task","preventionTips":["Never hold shared locks (DispatchCtx, ws_client) across long awaits in custom handlers","Use cancellation-aware patterns (tokio::select!) in any code that feeds the dispatch loop","Avoid blocking/CPU-heavy work on tokio workers so abort signals are observed promptly","Monitor for repeated recovery-failure logs and restart the node if a task is wedged"],"tags":["websocket","recovery","task-timeout","tokio","binance-futures"],"backgroundTag":"request-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}