What causes "old user stream dispatch task did not stop after abort" in nautechsystems/nautilus_trader?

Check what the old dispatch task is blocked on — usually a channel send or a lock in DispatchCtx — and make it cancellation-aware (tokio::select! on cancellation_token)

nautechsystems/nautilus_trader · error · anyhow::Error

old user stream dispatch task did not stop after abort

Error message

old user stream dispatch task did not stop after abort

What it means

During Binance Futures listen-key recovery, the old user-data dispatch task is aborted and given 2 seconds to be aborted and 2 seconds to be joined. If it neither completes nor aborts within those windows, finish_task reports TaskJoinOutcome::Incomplete and recovery bails so it can be retried with backoff, instead of spawning a second concurrent dispatcher on top of a stuck one.

Source

Thrown at crates/adapters/binance/src/futures/websocket/streams/recovery.rs:255

            .context("failed to close old user data WebSocket")?;
    }

    // Drain queued events from the old stream while the replacement buffers new events.
    let mut task_slot = ctx.ws_task.lock().await;
    if let Some(outcome) = finish_task(
        &mut task_slot,
        Duration::from_secs(2),
        Duration::from_secs(2),
    )
    .await
    {
        match outcome {
            TaskJoinOutcome::Completed(()) | TaskJoinOutcome::Aborted => {}
            TaskJoinOutcome::Failed(error) => {
                anyhow::bail!("old user stream dispatch task failed: {error}");
            }
            TaskJoinOutcome::Incomplete => {
                anyhow::bail!("old user stream dispatch task did not stop after abort");
            }
        }
    }

    let mut new_task = TaskSlot::new();
    new_task
        .spawn(run_user_stream_dispatch(
            new_stream,
            ctx.dispatch_ctx.clone(),
            ctx.recovery_tx.clone(),
            dispatch_fn,
        ))
        .map_err(|e| anyhow::anyhow!("failed to start recovered user stream dispatch task: {e}"))?;

    *ctx.ws_client.lock() = Some(new_ws);
    *task_slot = new_task;
    *ctx.listen_key.write() = Some(new_listen_key);
    *ctx.recovery_listen_key.write() = None;

View on GitHub (pinned to 18893faf8b)

Solutions

Check what the old dispatch task is blocked on — usually a channel send or a lock in DispatchCtx — and make it cancellation-aware (tokio::select! on cancellation_token)
Ensure no other task holds the DispatchCtx/shared locks for long periods so the dispatcher can observe abort
Increase tolerance by fixing runtime starvation: avoid blocking calls (std::thread::sleep, heavy CPU) on the tokio workers
If persistent, restart the node and report the stuck-task pattern to maintainers with logs

Example fix

// before: dispatcher blocks on send without observing cancellation
_tx.send(event).await;
// after: select on the abort signal
let _ = tokio::select! { _ = cancel.cancelled() => return, res = _tx.send(event) => res };

Defensive patterns

Strategy: retry

Validate before calling

// Users cannot pre-validate the internal task lifecycle; ensure no custom code
// holds DispatchCtx locks across awaits and avoid blocking calls on the runtime:
// e.g. never call std::thread::sleep inside handlers; use tokio::time::sleep

Try / catch

// Recovery retries automatically; on the ops side, alert if the same
// "did not stop after abort" message repeats across attempts:
// grep 'did not stop after abort' logs | wc -l  # >1 => investigate stuck task

Prevention

Never hold shared locks (DispatchCtx, ws_client) across long awaits in custom handlers
Use cancellation-aware patterns (tokio::select!) in any code that feeds the dispatch loop
Avoid blocking/CPU-heavy work on tokio workers so abort signals are observed promptly
Monitor for repeated recovery-failure logs and restart the node if a task is wedged

When it happens

Trigger: recovery is triggered (keepalive failure/expiry); the old dispatch task ignores the abort signal and stays alive past both finish_task timeouts — typically because it is blocked in a non-cancellation-aware await (e.g. a blocking send or a lock held by another holder).

Common situations: A downstream channel consumer is gone so the dispatcher blocks on an unbounded-channel send or receiver that never yields; a lock in DispatchCtx is held across a slow await; extremely slow/event-loop-starved runtime under heavy load.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/8d179798c89b6012. Report an issue: GitHub.