nautechsystems/nautilus_trader · error · anyhow::Error

old user stream dispatch task failed: {error}

Error message

old user stream dispatch task failed: {error}

What it means

During Binance Futures listen-key recovery, the adapter first aborts and joins the old user-data WebSocket dispatch task before spawning the replacement. If the joined task terminates with a panic or an error result instead of completing or being cleanly aborted, recover_user_data_stream bails with this message so recover_with_retry can retry the whole recovery with exponential backoff.

Source

Thrown at crates/adapters/binance/src/futures/websocket/streams/recovery.rs:252

        old_ws
            .close()
            .await
            .context("failed to close old user data WebSocket")?;
    }

    // Drain queued events from the old stream while the replacement buffers new events.
    let mut task_slot = ctx.ws_task.lock().await;
    if let Some(outcome) = finish_task(
        &mut task_slot,
        Duration::from_secs(2),
        Duration::from_secs(2),
    )
    .await
    {
        match outcome {
            TaskJoinOutcome::Completed(()) | TaskJoinOutcome::Aborted => {}
            TaskJoinOutcome::Failed(error) => {
                anyhow::bail!("old user stream dispatch task failed: {error}");
            }
            TaskJoinOutcome::Incomplete => {
                anyhow::bail!("old user stream dispatch task did not stop after abort");
            }
        }
    }

    let mut new_task = TaskSlot::new();
    new_task
        .spawn(run_user_stream_dispatch(
            new_stream,
            ctx.dispatch_ctx.clone(),
            ctx.recovery_tx.clone(),
            dispatch_fn,
        ))
        .map_err(|e| anyhow::anyhow!("failed to start recovered user stream dispatch task: {e}"))?;

    *ctx.ws_client.lock() = Some(new_ws);

View on GitHub (pinned to 18893faf8b)

Solutions

1. Inspect the embedded {error} in the message to find the dispatch task's underlying failure and fix that root cause first
2. Retry the connection: recovery already retries indefinitely with exponential backoff, so a transient panic usually self-heals — verify the next attempt logs success
3. Upgrade/patch the adapter if the dispatch closure panics deterministically on a specific message type; capture the message and file an issue
4. Check system resources (fd limits, memory) if panics correlate with load or long-running sessions

Example fix

// before: dispatch loop can panic on malformed payload
let event = parse(msg).unwrap();
// after: handle parse failure inside the dispatch task so the join outcome is clean
let event = match parse(msg) { Ok(e) => e, Err(e) => { log::error!("dispatch parse failed: {e}"); continue; } };

// Recovery is internal; users cannot pre-validate. Ensure infrastructure health:
// check fd limits and runtime health before long sessions
ulimit -n  # ensure ample file descriptors for WebSocket connections

// Errors here are logged by the internal retry loop; on the node side, alert on
// repeated "Listen key recovery attempt N failed" log lines:
// if log.matches("old user stream dispatch task failed").count > 3 { page_oncall() }

When it happens

Trigger: A listen-key keepalive failure or expiry triggers recovery; while draining the old dispatch task via finish_task, the task's JoinHandle resolves to TaskJoinOutcome::Failed(error) — i.e. the old dispatch task panicked or returned Err rather than finishing within the 2s abort/join windows.

Common situations: Network stalls or a poisoned WebSocket stream causing the dispatch loop to error out mid-recovery; a panic inside the message-dispatch closure (e.g. a bug in event handling or downstream channel send); resource exhaustion on the node running the trader.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• old user stream dispatch task did not stop after abort
• failed to start recovered user stream dispatch task: {e}
• failed to connect Binance Futures private WebSocket
• failed to finish Binance Futures data command tasks: {e}
• BinanceFuturesWebSocketClient requires UsdM or CoinM product

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/02cfa226cf11eac9. Report an issue: GitHub.