{"record":{"id":"02cfa226cf11eac9","repo":"nautechsystems/nautilus_trader","slug":"old-user-stream-dispatch-task-failed-error","errorCode":null,"errorMessage":"old user stream dispatch task failed: {error}","messagePattern":"old user stream dispatch task failed: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/binance/src/futures/websocket/streams/recovery.rs","lineNumber":252,"sourceCode":"        old_ws\n            .close()\n            .await\n            .context(\"failed to close old user data WebSocket\")?;\n    }\n\n    // Drain queued events from the old stream while the replacement buffers new events.\n    let mut task_slot = ctx.ws_task.lock().await;\n    if let Some(outcome) = finish_task(\n        &mut task_slot,\n        Duration::from_secs(2),\n        Duration::from_secs(2),\n    )\n    .await\n    {\n        match outcome {\n            TaskJoinOutcome::Completed(()) | TaskJoinOutcome::Aborted => {}\n            TaskJoinOutcome::Failed(error) => {\n                anyhow::bail!(\"old user stream dispatch task failed: {error}\");\n            }\n            TaskJoinOutcome::Incomplete => {\n                anyhow::bail!(\"old user stream dispatch task did not stop after abort\");\n            }\n        }\n    }\n\n    let mut new_task = TaskSlot::new();\n    new_task\n        .spawn(run_user_stream_dispatch(\n            new_stream,\n            ctx.dispatch_ctx.clone(),\n            ctx.recovery_tx.clone(),\n            dispatch_fn,\n        ))\n        .map_err(|e| anyhow::anyhow!(\"failed to start recovered user stream dispatch task: {e}\"))?;\n\n    *ctx.ws_client.lock() = Some(new_ws);","sourceCodeStart":234,"sourceCodeEnd":270,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/binance/src/futures/websocket/streams/recovery.rs#L234-L270","documentation":"During Binance Futures listen-key recovery, the adapter first aborts and joins the old user-data WebSocket dispatch task before spawning the replacement. If the joined task terminates with a panic or an error result instead of completing or being cleanly aborted, recover_user_data_stream bails with this message so recover_with_retry can retry the whole recovery with exponential backoff.","triggerScenarios":"A listen-key keepalive failure or expiry triggers recovery; while draining the old dispatch task via finish_task, the task's JoinHandle resolves to TaskJoinOutcome::Failed(error) — i.e. the old dispatch task panicked or returned Err rather than finishing within the 2s abort/join windows.","commonSituations":"Network stalls or a poisoned WebSocket stream causing the dispatch loop to error out mid-recovery; a panic inside the message-dispatch closure (e.g. a bug in event handling or downstream channel send); resource exhaustion on the node running the trader.","solutions":["Inspect the embedded {error} in the message to find the dispatch task's underlying failure and fix that root cause first","Retry the connection: recovery already retries indefinitely with exponential backoff, so a transient panic usually self-heals — verify the next attempt logs success","Upgrade/patch the adapter if the dispatch closure panics deterministically on a specific message type; capture the message and file an issue","Check system resources (fd limits, memory) if panics correlate with load or long-running sessions"],"exampleFix":"// before: dispatch loop can panic on malformed payload\nlet event = parse(msg).unwrap();\n// after: handle parse failure inside the dispatch task so the join outcome is clean\nlet event = match parse(msg) { Ok(e) => e, Err(e) => { log::error!(\"dispatch parse failed: {e}\"); continue; } };","handlingStrategy":"retry","validationCode":"// Recovery is internal; users cannot pre-validate. Ensure infrastructure health:\n// check fd limits and runtime health before long sessions\nulimit -n  # ensure ample file descriptors for WebSocket connections","typeGuard":null,"tryCatchPattern":"// Errors here are logged by the internal retry loop; on the node side, alert on\n// repeated \"Listen key recovery attempt N failed\" log lines:\n// if log.matches(\"old user stream dispatch task failed\").count > 3 { page_oncall() }","preventionTips":["Keep the node's runtime and resources healthy (memory, fd limits) to avoid dispatch-task panics","Monitor recovery logs and alert on repeated failures","Pin known-good adapter versions and read changelogs for dispatch-loop fixes","Test long-running sessions in paper mode before live"],"tags":["websocket","recovery","task-join","tokio","binance-futures"],"backgroundTag":"internal-invariant-violation","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}