What causes "Failed to terminate AX execution session tasks: {e}" in nautechsystems/nautilus_trader?

Inspect the wrapped inner error for which session tasks did not exit.

Failed to terminate AX execution session tasks: {e}

Error message

Failed to terminate AX execution session tasks: {e}

What it means

Raised in `await_session_tasks` when the execution client's session-task supervisor (`session_tasks`) fails to finish shutdown within its 1s grace and 2s hard deadline. Session tasks (e.g., session keep-alive/streams for the AX execution client) did not terminate cleanly during teardown.

Source

Thrown at crates/adapters/architect_ax/src/execution.rs:425

        self.session_tasks.begin_shutdown();
        self.ws_orders.begin_shutdown();
    }

    async fn await_pending_tasks(&self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.pending_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate AX execution tasks: {e}"))?;
        Ok(())
    }

    async fn await_session_tasks(&self) -> anyhow::Result<()> {
        self.session_tasks.begin_shutdown();
        self.session_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate AX execution session tasks: {e}"))?;
        Ok(())
    }

    async fn teardown_partial_connect(&mut self) -> anyhow::Result<()> {
        self.abort_session_tasks();
        self.abort_pending_tasks();
        self.http_client.cancel_all_requests();

        if let Err(e) = self.ws_orders.close().await {
            self.shutdown_errors
                .push(format!("AX orders WebSocket shutdown failed: {e}"));
        }

        let (session_result, pending_result) =
            tokio::join!(self.await_session_tasks(), self.await_pending_tasks());
        self.core.set_disconnected();

        if let Err(e) = session_result {

View on GitHub (pinned to 18893faf8b)

Solutions

Inspect the wrapped inner error for which session tasks did not exit.
Ensure session task loops select on the shutdown/cancellation signal and exit promptly.
Add timeouts to session network reads so tasks cannot block indefinitely.
Increase the grace/timeout durations if sessions need more time to wind down.
Verify a fallback abort path (e.g., `abort_session_tasks`) is invoked after the failure to avoid leaked tasks.

Example fix

// before
loop {
    let msg = ws.next().await;  // blocks shutdown
    handle(msg);
}
// after
loop {
    tokio::select! {
        msg = ws.next() => handle(msg),
        _ = shutdown.cancelled() => break,
    }
}

Defensive patterns

Strategy: try-catch

Try / catch

match client.await_session_tasks().await {
    Err(e) => {
        log::warn!("session tasks shutdown timed out: {e:#}");
        client.abort_session_tasks(); // forced fallback
    }
    Ok(()) => {},
}

Prevention

Structure session loops with tokio::select! over the shutdown signal.
Set read/write timeouts on session sockets so loops cannot block indefinitely.
Stop reconnect loops before initiating shutdown.
Always run a forced abort path after graceful session shutdown fails.

When it happens

Trigger: Calling `disconnect()`/`await_session_tasks` when session tasks remain blocked — typically a WebSocket session loop or stream reader stuck on a hung connection that ignores the shutdown signal until the hard deadline passes.

Common situations: WS session loop not observing the cancellation token; TCP connection half-open during teardown; slow AX server responses keeping session handlers alive; shutdown racing with an active reconnect loop.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/c7474e038bb1aa96. Report an issue: GitHub.