{"record":{"id":"80e996cfbf97e2a3","repo":"nautechsystems/nautilus_trader","slug":"failed-to-start-recovered-user-stream-dispatch-tas","errorCode":null,"errorMessage":"failed to start recovered user stream dispatch task: {e}","messagePattern":"failed to start recovered user stream dispatch task: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/binance/src/futures/websocket/streams/recovery.rs","lineNumber":268,"sourceCode":"            TaskJoinOutcome::Completed(()) | TaskJoinOutcome::Aborted => {}\n            TaskJoinOutcome::Failed(error) => {\n                anyhow::bail!(\"old user stream dispatch task failed: {error}\");\n            }\n            TaskJoinOutcome::Incomplete => {\n                anyhow::bail!(\"old user stream dispatch task did not stop after abort\");\n            }\n        }\n    }\n\n    let mut new_task = TaskSlot::new();\n    new_task\n        .spawn(run_user_stream_dispatch(\n            new_stream,\n            ctx.dispatch_ctx.clone(),\n            ctx.recovery_tx.clone(),\n            dispatch_fn,\n        ))\n        .map_err(|e| anyhow::anyhow!(\"failed to start recovered user stream dispatch task: {e}\"))?;\n\n    *ctx.ws_client.lock() = Some(new_ws);\n    *task_slot = new_task;\n    *ctx.listen_key.write() = Some(new_listen_key);\n    *ctx.recovery_listen_key.write() = None;\n\n    Ok(())\n}\n\nasync fn close_recovery_listen_key(ctx: &RecoveryCtx) -> anyhow::Result<()> {\n    let key = ctx.recovery_listen_key.read().clone();\n    let Some(key) = key else {\n        return Ok(());\n    };\n\n    ctx.http_client\n        .close_listen_key(key.expose_secret())\n        .await","sourceCodeStart":250,"sourceCodeEnd":286,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/binance/src/futures/websocket/streams/recovery.rs#L250-L286","documentation":"After a successful reconnect, `recover_user_data_stream` spawns `run_user_stream_dispatch` to resume processing messages from the new private stream. This error wraps a spawn failure from the tokio runtime (e.g. the runtime is shutting down or has no worker capacity), so the recovered stream would be connected but never dispatched, and the recovery is failed so `recover_with_retry` retries.","triggerScenarios":"`recover_with_retry` → `recover_user_data_stream` → `tokio::spawn(...).map_err(...)` returns `JoinError`/spawn failure — practically always during application shutdown when the runtime is being dropped, or if the task was cancelled immediately.","commonSituations":"Live node shutting down exactly while a listen-key recovery is in flight; runtime misconfiguration with a tiny/terminated runtime; panic in the spawned dispatch task surfacing as JoinError.","solutions":["If this appears at shutdown, it is benign — ensure recovery is aborted/cancelled gracefully during node teardown before the runtime drops.","Check whether the dispatch task panicked by looking for an accompanying JoinError panic message in `e`.","Ensure the tokio runtime outlives recovery attempts (keep the runtime handle alive for the trader's lifetime).","Retry — `recover_with_retry` will schedule another recovery attempt on this error."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Treat spawn failure during shutdown as benign; otherwise retry recovery\nif let Err(e) = recovery_result {\n    if e.to_string().contains(\"failed to start recovered user stream dispatch task\") {\n        log::warn!(\"dispatch task spawn failed: {e}; will retry or shut down\");\n    }\n}","preventionTips":["Cancel recovery tasks explicitly during node shutdown before dropping the runtime","Keep the tokio runtime alive for the full trader lifecycle","Check for panics in dispatch tasks if JoinErrors recur"],"tags":["tokio","task-spawn","websocket","binance-futures","recovery"],"backgroundTag":"network-request-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}