{"record":{"id":"0254a01f3c352859","repo":"nautechsystems/nautilus_trader","slug":"failed-to-terminate-ax-data-tasks-e","errorCode":null,"errorMessage":"Failed to terminate AX data tasks: {e}","messagePattern":"Failed to terminate AX data tasks: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"crates/adapters/architect_ax/src/data.rs","lineNumber":364,"sourceCode":"        self.ws_client.begin_shutdown();\n\n        for cancellation in self.funding_rate_cancellations.values() {\n            cancellation.cancel();\n        }\n    }\n\n    async fn finish_all_tasks(&mut self) -> anyhow::Result<()> {\n        self.pending_tasks.begin_shutdown();\n        self.session_tasks.begin_shutdown();\n        let (pending_result, session_result) = tokio::join!(\n            self.pending_tasks\n                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)),\n            self.session_tasks\n                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)),\n        );\n        self.funding_rate_cancellations.clear();\n\n        pending_result.map_err(|e| anyhow::anyhow!(\"Failed to terminate AX data tasks: {e}\"))?;\n        session_result\n            .map_err(|e| anyhow::anyhow!(\"Failed to terminate AX data session tasks: {e}\"))?;\n        Ok(())\n    }\n\n    async fn teardown_partial_connect(&mut self) -> anyhow::Result<()> {\n        self.abort_all_tasks();\n\n        if let Err(e) = self.ws_client.close().await {\n            self.shutdown_errors.push(e.to_string());\n        }\n\n        if let Err(e) = self.finish_all_tasks().await {\n            self.shutdown_errors.push(e.to_string());\n        }\n        self.is_connected.store(false, Ordering::Release);\n\n        if !self.shutdown_errors.is_empty() {","sourceCodeStart":346,"sourceCodeEnd":382,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/architect_ax/src/data.rs#L346-L382","documentation":"During shutdown, finish_all_tasks awaits pending tasks' graceful finish_shutdown (1s finish / 2s hard timeouts). If any pending task fails to terminate in that window, the error is wrapped as \"Failed to terminate AX data tasks\" and disconnect/teardown reports it. It signals incomplete cleanup of streaming tasks.","triggerScenarios":"Calling disconnect (or teardown_partial_connect after a failed connect) while pending data tasks (quote/trade/bar streams) are stuck and do not finish within the 1s/2s shutdown timeouts.","commonSituations":"Slow or hung network socket preventing task completion; tasks blocked awaiting messages on a stalled connection; unresponsive WS server during shutdown.","solutions":["Check network connectivity and whether the AX WS connection is stalled; retry disconnect after the connection recovers","Abort tasks rather than awaiting graceful shutdown if timeouts keep firing (see abort_all_tasks path)","Increase the finish_shutdown durations if streams legitimately need longer to wind down","Investigate why individual tasks hang (e.g. missing cancellation token triggers)"],"exampleFix":"// before\nself.pending_tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n// after\nself.pending_tasks.abort_all(); // force-abort stuck tasks when graceful shutdown times out\nself.pending_tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))","handlingStrategy":"try-catch","validationCode":"// Rust\n// before disconnect: ensure tasks were given a cancellation signal\n// if !pending_tasks.is_empty() { cancellation_token.cancel(); }","typeGuard":null,"tryCatchPattern":"// Rust\nif let Err(e) = client.disconnect().await {\n    if e.to_string().contains(\"Failed to terminate AX data tasks\") {\n        log::warn!(\"graceful shutdown timed out; forcing teardown\");\n        // proceed — resources will be dropped with the client\n    }\n}","preventionTips":["Cancel the client's cancellation token before disconnecting to let tasks wind down","Avoid disconnecting while streams are mid-write on a stalled socket","Treat this as a warning: the client is being dropped anyway; log and continue","Investigate recurring occurrences as a sign of hung tasks not honoring cancellation"],"tags":["shutdown","tasks","timeout","websocket"],"backgroundTag":"request-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}