{"record":{"id":"dc2650e233f339e2","repo":"nautechsystems/nautilus_trader","slug":"failed-to-terminate-ax-execution-tasks-e","errorCode":null,"errorMessage":"Failed to terminate AX execution tasks: {e}","messagePattern":"Failed to terminate AX execution tasks: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/architect_ax/src/execution.rs","lineNumber":416,"sourceCode":"            log::warn!(\"Skipping AX {description} after shutdown began: {e}\");\n        }\n    }\n\n    fn abort_pending_tasks(&self) {\n        self.pending_tasks.begin_shutdown();\n    }\n\n    fn abort_session_tasks(&self) {\n        self.session_tasks.begin_shutdown();\n        self.ws_orders.begin_shutdown();\n    }\n\n    async fn await_pending_tasks(&self) -> anyhow::Result<()> {\n        self.pending_tasks.begin_shutdown();\n        self.pending_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to terminate AX execution tasks: {e}\"))?;\n        Ok(())\n    }\n\n    async fn await_session_tasks(&self) -> anyhow::Result<()> {\n        self.session_tasks.begin_shutdown();\n        self.session_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to terminate AX execution session tasks: {e}\"))?;\n        Ok(())\n    }\n\n    async fn teardown_partial_connect(&mut self) -> anyhow::Result<()> {\n        self.abort_session_tasks();\n        self.abort_pending_tasks();\n        self.http_client.cancel_all_requests();\n\n        if let Err(e) = self.ws_orders.close().await {","sourceCodeStart":398,"sourceCodeEnd":434,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/architect_ax/src/execution.rs#L398-L434","documentation":"Raised in `await_pending_tasks` when the execution client's pending-task supervisor (`pending_tasks`) fails to complete its shutdown within the configured grace (1s) and hard (2s) deadlines. It is a teardown-time error indicating AX execution order/task workers did not terminate cleanly.","triggerScenarios":"Calling `disconnect()`/`await_pending_tasks` during client shutdown when spawned pending tasks are stuck (blocked on a hung HTTP/WS call, deadlocked, or ignoring cancellation) and exceed the 1s grace and 2s timeout windows.","commonSituations":"Network hang during shutdown causing in-flight request tasks to block past the timeout; slow/stalled AX API responses during teardown; shutdown races where tasks await a channel that never closes.","solutions":["Inspect the wrapped inner error to see whether tasks timed out or were already aborted.","Ensure in-flight requests have reasonable HTTP/WS timeouts so tasks can observe cancellation.","Increase the shutdown grace/timeout durations if legitimate work needs more than 1-2s to finish.","Check for deadlocks in task code (e.g., awaiting a channel that is never closed).","If purely shutdown-time and non-data-loss, treat as a logged warning and verify tasks are forcibly aborted afterwards."],"exampleFix":"// before\nself.pending_tasks\n    .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n    .await?;\n// after\nself.pending_tasks\n    .finish_shutdown(Duration::from_secs(5), Duration::from_secs(10))\n    .await?;","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"match client.await_pending_tasks().await {\n    Err(e) => log::warn!(\"pending tasks did not shut down cleanly (forced abort follows): {e:#}\"),\n    Ok(()) => {},\n}\nclient.abort_pending_tasks();","preventionTips":["Give every spawned task a cancellation path and bounded network timeouts.","Size shutdown grace/timeout windows to realistic worst-case request latency.","Always invoke the abort fallback after a failed graceful shutdown to prevent task leaks.","Log in-flight task counts at disconnect start to spot stuck tasks early."],"tags":["shutdown","tasks","timeout","teardown"],"backgroundTag":"task-shutdown-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}