{"record":{"id":"974b758152e69d5d","repo":"nautechsystems/nautilus_trader","slug":"blockchain-process-task-shutdown-failed-e","errorCode":null,"errorMessage":"Blockchain process task shutdown failed: {e}","messagePattern":"Blockchain process task shutdown failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/data/client.rs","lineNumber":1212,"sourceCode":"\n                Ok(())\n            }\n        }\n    }\n\n    /// Waits for the background processing task to complete.\n    ///\n    /// This method blocks until the spawned process task finishes execution,\n    /// which typically happens after a shutdown signal is sent.\n    ///\n    /// # Errors\n    ///\n    /// Returns an error when bounded task shutdown fails.\n    pub async fn await_process_task_close(&self) -> anyhow::Result<()> {\n        self.session_tasks\n            .finish_shutdown(Duration::from_secs(2), Duration::from_secs(2))\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Blockchain process task shutdown failed: {e}\"))?;\n        Ok(())\n    }\n\n    async fn prepare_task_group(&mut self) -> anyhow::Result<()> {\n        if !self.session_tasks.is_open() {\n            self.session_tasks.begin_shutdown();\n            self.await_process_task_close().await?;\n            self.reset_channels();\n            self.session_tasks\n                .start_generation()\n                .map_err(|e| anyhow::anyhow!(\"Failed to start blockchain task generation: {e}\"))?;\n            self.cancellation_token = self.session_tasks.cancellation_token();\n        }\n        Ok(())\n    }\n\n    fn reset_channels(&mut self) {\n        let (hypersync_tx, hypersync_rx) = tokio::sync::mpsc::unbounded_channel();","sourceCodeStart":1194,"sourceCodeEnd":1230,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/data/client.rs#L1194-L1230","documentation":"await_process_task_close wraps a failure of session_tasks.finish_shutdown (with 2s grace and 2s await timeouts) into an anyhow error. It means the blockchain data client's background process tasks did not shut down cleanly within the bounded window — e.g. tasks stuck in blocking I/O, network calls not honoring the cancellation token, or the runtime under load.","triggerScenarios":"Calling disconnect() (or connect()/prepare_task_group cleanup) while long-running WebSocket/RPC streaming tasks ignore the cancellation token and exceed the 2-second shutdown budget.","commonSituations":"Slow or hung RPC/WS connections to remote blockchain nodes; tasks in blocking (non-async) code paths; heavy system load starving the tokio runtime during teardown; repeated connect/disconnect cycles not awaiting prior shutdown.","solutions":["Inspect task logs to identify which session task ignores shutdown; ensure it selects on the cancellation token.","Increase the shutdown timeouts in finish_shutdown if tasks legitimately need more than 2s to unwind.","Retry disconnect after a short delay; some stuck I/O resolves once the peer socket errors out.","Verify network reachability of RPC/WS endpoints — hung sockets are the most common cause of shutdown hangs."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// retry with backoff\nfor attempt in 0..3 {\n    match client.disconnect().await {\n        Ok(()) => break,\n        Err(e) if e.to_string().contains(\"shutdown failed\") => {\n            tokio::time::sleep(Duration::from_millis(500 * (attempt + 1))).await;\n        }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Ensure all spawned session tasks select on the cancellation token so they exit promptly","Avoid blocking (std::thread-style) work inside background tasks","Check RPC/WS endpoint health; hung sockets are the usual shutdown hang cause","Await full disconnect before reconnecting"],"tags":["tokio","shutdown","task-management","blockchain"],"backgroundTag":"request-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}