nautechsystems/nautilus_trader · error

Blockchain process task shutdown failed: {e}

Error message

Blockchain process task shutdown failed: {e}

What it means

await_process_task_close wraps a failure of session_tasks.finish_shutdown (with 2s grace and 2s await timeouts) into an anyhow error. It means the blockchain data client's background process tasks did not shut down cleanly within the bounded window — e.g. tasks stuck in blocking I/O, network calls not honoring the cancellation token, or the runtime under load.

Source

Thrown at crates/adapters/blockchain/src/data/client.rs:1212

                Ok(())
            }
        }
    }

    /// Waits for the background processing task to complete.
    ///
    /// This method blocks until the spawned process task finishes execution,
    /// which typically happens after a shutdown signal is sent.
    ///
    /// # Errors
    ///
    /// Returns an error when bounded task shutdown fails.
    pub async fn await_process_task_close(&self) -> anyhow::Result<()> {
        self.session_tasks
            .finish_shutdown(Duration::from_secs(2), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Blockchain process task shutdown failed: {e}"))?;
        Ok(())
    }

    async fn prepare_task_group(&mut self) -> anyhow::Result<()> {
        if !self.session_tasks.is_open() {
            self.session_tasks.begin_shutdown();
            self.await_process_task_close().await?;
            self.reset_channels();
            self.session_tasks
                .start_generation()
                .map_err(|e| anyhow::anyhow!("Failed to start blockchain task generation: {e}"))?;
            self.cancellation_token = self.session_tasks.cancellation_token();
        }
        Ok(())
    }

    fn reset_channels(&mut self) {
        let (hypersync_tx, hypersync_rx) = tokio::sync::mpsc::unbounded_channel();

View on GitHub (pinned to 18893faf8b)

Solutions

1. Inspect task logs to identify which session task ignores shutdown; ensure it selects on the cancellation token.
2. Increase the shutdown timeouts in finish_shutdown if tasks legitimately need more than 2s to unwind.
3. Retry disconnect after a short delay; some stuck I/O resolves once the peer socket errors out.
4. Verify network reachability of RPC/WS endpoints — hung sockets are the most common cause of shutdown hangs.

// retry with backoff
for attempt in 0..3 {
    match client.disconnect().await {
        Ok(()) => break,
        Err(e) if e.to_string().contains("shutdown failed") => {
            tokio::time::sleep(Duration::from_millis(500 * (attempt + 1))).await;
        }
        Err(e) => return Err(e),
    }
}

When it happens

Trigger: Calling disconnect() (or connect()/prepare_task_group cleanup) while long-running WebSocket/RPC streaming tasks ignore the cancellation token and exceed the 2-second shutdown budget.

Common situations: Slow or hung RPC/WS connections to remote blockchain nodes; tasks in blocking (non-async) code paths; heavy system load starving the tokio runtime during teardown; repeated connect/disconnect cycles not awaiting prior shutdown.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• failed to finish Binance Futures data command tasks: {e}
• failed to finish Binance Spot data command tasks: {e}
• Hyperliquid WebSocket handler failed: {error}
• Hyperliquid WebSocket handler did not stop after abort
• {context}: {}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/974b758152e69d5d. Report an issue: GitHub.