What causes "Failed to terminate AX execution tasks: {e}" in nautechsystems/nautilus_trader?

Inspect the wrapped inner error to see whether tasks timed out or were already aborted.

Failed to terminate AX execution tasks: {e}

Error message

Failed to terminate AX execution tasks: {e}

What it means

Raised in `await_pending_tasks` when the execution client's pending-task supervisor (`pending_tasks`) fails to complete its shutdown within the configured grace (1s) and hard (2s) deadlines. It is a teardown-time error indicating AX execution order/task workers did not terminate cleanly.

Source

Thrown at crates/adapters/architect_ax/src/execution.rs:416

            log::warn!("Skipping AX {description} after shutdown began: {e}");
        }
    }

    fn abort_pending_tasks(&self) {
        self.pending_tasks.begin_shutdown();
    }

    fn abort_session_tasks(&self) {
        self.session_tasks.begin_shutdown();
        self.ws_orders.begin_shutdown();
    }

    async fn await_pending_tasks(&self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.pending_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate AX execution tasks: {e}"))?;
        Ok(())
    }

    async fn await_session_tasks(&self) -> anyhow::Result<()> {
        self.session_tasks.begin_shutdown();
        self.session_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate AX execution session tasks: {e}"))?;
        Ok(())
    }

    async fn teardown_partial_connect(&mut self) -> anyhow::Result<()> {
        self.abort_session_tasks();
        self.abort_pending_tasks();
        self.http_client.cancel_all_requests();

        if let Err(e) = self.ws_orders.close().await {

View on GitHub (pinned to 18893faf8b)

Solutions

Inspect the wrapped inner error to see whether tasks timed out or were already aborted.
Ensure in-flight requests have reasonable HTTP/WS timeouts so tasks can observe cancellation.
Increase the shutdown grace/timeout durations if legitimate work needs more than 1-2s to finish.
Check for deadlocks in task code (e.g., awaiting a channel that is never closed).
If purely shutdown-time and non-data-loss, treat as a logged warning and verify tasks are forcibly aborted afterwards.

Example fix

// before
self.pending_tasks
    .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
    .await?;
// after
self.pending_tasks
    .finish_shutdown(Duration::from_secs(5), Duration::from_secs(10))
    .await?;

Defensive patterns

Strategy: try-catch

Try / catch

match client.await_pending_tasks().await {
    Err(e) => log::warn!("pending tasks did not shut down cleanly (forced abort follows): {e:#}"),
    Ok(()) => {},
}
client.abort_pending_tasks();

Prevention

Give every spawned task a cancellation path and bounded network timeouts.
Size shutdown grace/timeout windows to realistic worst-case request latency.
Always invoke the abort fallback after a failed graceful shutdown to prevent task leaks.
Log in-flight task counts at disconnect start to spot stuck tasks early.

When it happens

Trigger: Calling `disconnect()`/`await_pending_tasks` during client shutdown when spawned pending tasks are stuck (blocked on a hung HTTP/WS call, deadlocked, or ignoring cancellation) and exceed the 1s grace and 2s timeout windows.

Common situations: Network hang during shutdown causing in-flight request tasks to block past the timeout; slow/stalled AX API responses during teardown; shutdown races where tasks await a channel that never closes.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/dc2650e233f339e2. Report an issue: GitHub.