nautechsystems/nautilus_trader · error

failed to start recovered user stream dispatch task: {e}

Error message

failed to start recovered user stream dispatch task: {e}

What it means

After a successful reconnect, `recover_user_data_stream` spawns `run_user_stream_dispatch` to resume processing messages from the new private stream. This error wraps a spawn failure from the tokio runtime (e.g. the runtime is shutting down or has no worker capacity), so the recovered stream would be connected but never dispatched, and the recovery is failed so `recover_with_retry` retries.

Source

Thrown at crates/adapters/binance/src/futures/websocket/streams/recovery.rs:268

            TaskJoinOutcome::Completed(()) | TaskJoinOutcome::Aborted => {}
            TaskJoinOutcome::Failed(error) => {
                anyhow::bail!("old user stream dispatch task failed: {error}");
            }
            TaskJoinOutcome::Incomplete => {
                anyhow::bail!("old user stream dispatch task did not stop after abort");
            }
        }
    }

    let mut new_task = TaskSlot::new();
    new_task
        .spawn(run_user_stream_dispatch(
            new_stream,
            ctx.dispatch_ctx.clone(),
            ctx.recovery_tx.clone(),
            dispatch_fn,
        ))
        .map_err(|e| anyhow::anyhow!("failed to start recovered user stream dispatch task: {e}"))?;

    *ctx.ws_client.lock() = Some(new_ws);
    *task_slot = new_task;
    *ctx.listen_key.write() = Some(new_listen_key);
    *ctx.recovery_listen_key.write() = None;

    Ok(())
}

async fn close_recovery_listen_key(ctx: &RecoveryCtx) -> anyhow::Result<()> {
    let key = ctx.recovery_listen_key.read().clone();
    let Some(key) = key else {
        return Ok(());
    };

    ctx.http_client
        .close_listen_key(key.expose_secret())
        .await

View on GitHub (pinned to 18893faf8b)

Solutions

1. If this appears at shutdown, it is benign — ensure recovery is aborted/cancelled gracefully during node teardown before the runtime drops.
2. Check whether the dispatch task panicked by looking for an accompanying JoinError panic message in `e`.
3. Ensure the tokio runtime outlives recovery attempts (keep the runtime handle alive for the trader's lifetime).
4. Retry — `recover_with_retry` will schedule another recovery attempt on this error.

// Treat spawn failure during shutdown as benign; otherwise retry recovery
if let Err(e) = recovery_result {
    if e.to_string().contains("failed to start recovered user stream dispatch task") {
        log::warn!("dispatch task spawn failed: {e}; will retry or shut down");
    }
}

When it happens

Trigger: `recover_with_retry` → `recover_user_data_stream` → `tokio::spawn(...).map_err(...)` returns `JoinError`/spawn failure — practically always during application shutdown when the runtime is being dropped, or if the task was cancelled immediately.

Common situations: Live node shutting down exactly while a listen-key recovery is in flight; runtime misconfiguration with a tiny/terminated runtime; panic in the spawned dispatch task surfacing as JoinError.

Related errors

• old user stream dispatch task failed: {error}
• old user stream dispatch task did not stop after abort
• Failed to start Hyperliquid WebSocket handler task: {e}
• failed to start user stream dispatch task: {e}
• failed to connect Binance Futures private WebSocket

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/80e996cfbf97e2a3. Report an issue: GitHub.