nautechsystems/nautilus_trader · warning

Failed to terminate AX data tasks: {e}

Error message

Failed to terminate AX data tasks: {e}

What it means

During shutdown, finish_all_tasks awaits pending tasks' graceful finish_shutdown (1s finish / 2s hard timeouts). If any pending task fails to terminate in that window, the error is wrapped as "Failed to terminate AX data tasks" and disconnect/teardown reports it. It signals incomplete cleanup of streaming tasks.

Source

Thrown at crates/adapters/architect_ax/src/data.rs:364

        self.ws_client.begin_shutdown();

        for cancellation in self.funding_rate_cancellations.values() {
            cancellation.cancel();
        }
    }

    async fn finish_all_tasks(&mut self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.session_tasks.begin_shutdown();
        let (pending_result, session_result) = tokio::join!(
            self.pending_tasks
                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)),
            self.session_tasks
                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)),
        );
        self.funding_rate_cancellations.clear();

        pending_result.map_err(|e| anyhow::anyhow!("Failed to terminate AX data tasks: {e}"))?;
        session_result
            .map_err(|e| anyhow::anyhow!("Failed to terminate AX data session tasks: {e}"))?;
        Ok(())
    }

    async fn teardown_partial_connect(&mut self) -> anyhow::Result<()> {
        self.abort_all_tasks();

        if let Err(e) = self.ws_client.close().await {
            self.shutdown_errors.push(e.to_string());
        }

        if let Err(e) = self.finish_all_tasks().await {
            self.shutdown_errors.push(e.to_string());
        }
        self.is_connected.store(false, Ordering::Release);

        if !self.shutdown_errors.is_empty() {

View on GitHub (pinned to 18893faf8b)

Solutions

1. Check network connectivity and whether the AX WS connection is stalled; retry disconnect after the connection recovers
2. Abort tasks rather than awaiting graceful shutdown if timeouts keep firing (see abort_all_tasks path)
3. Increase the finish_shutdown durations if streams legitimately need longer to wind down
4. Investigate why individual tasks hang (e.g. missing cancellation token triggers)

Example fix

// before
self.pending_tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
// after
self.pending_tasks.abort_all(); // force-abort stuck tasks when graceful shutdown times out
self.pending_tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))

// Rust
// before disconnect: ensure tasks were given a cancellation signal
// if !pending_tasks.is_empty() { cancellation_token.cancel(); }

// Rust
if let Err(e) = client.disconnect().await {
    if e.to_string().contains("Failed to terminate AX data tasks") {
        log::warn!("graceful shutdown timed out; forcing teardown");
        // proceed — resources will be dropped with the client
    }
}

When it happens

Trigger: Calling disconnect (or teardown_partial_connect after a failed connect) while pending data tasks (quote/trade/bar streams) are stuck and do not finish within the 1s/2s shutdown timeouts.

Common situations: Slow or hung network socket preventing task completion; tasks blocked awaiting messages on a stalled connection; unresponsive WS server during shutdown.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Failed to terminate AX data session tasks: {e}
• Hyperliquid WebSocket handler did not stop after abort
• Failed to terminate AX execution tasks: {e}
• Failed to terminate AX execution session tasks: {e}
• Architect AX data WebSocket handler did not stop after abort

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/0254a01f3c352859. Report an issue: GitHub.