{"record":{"id":"5c9f9b0d25d4f706","repo":"nautechsystems/nautilus_trader","slug":"failed-to-terminate-hyperliquid-data-tasks-e","errorCode":null,"errorMessage":"Failed to terminate Hyperliquid data tasks: {e}","messagePattern":"Failed to terminate Hyperliquid data tasks: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"warning","filePath":"crates/adapters/hyperliquid/src/data.rs","lineNumber":260,"sourceCode":"\n        if let Err(e) = self.await_pending_tasks().await {\n            self.shutdown_errors.push(e.to_string());\n        }\n        self.clear_stream_health();\n        self.is_connected.store(false, Ordering::Release);\n\n        if !self.shutdown_errors.is_empty() {\n            anyhow::bail!(std::mem::take(&mut self.shutdown_errors).join(\"; \"));\n        }\n        Ok(())\n    }\n\n    async fn await_pending_tasks(&self) -> anyhow::Result<()> {\n        self.pending_tasks.begin_shutdown();\n        self.pending_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to terminate Hyperliquid data tasks: {e}\"))?;\n        Ok(())\n    }\n\n    async fn await_session_tasks(&self) -> anyhow::Result<()> {\n        self.session_tasks.begin_shutdown();\n        self.session_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| {\n                anyhow::anyhow!(\"Failed to terminate Hyperliquid data session tasks: {e}\")\n            })?;\n        Ok(())\n    }\n\n    fn clear_stream_health(&self) {\n        self.stream_health.lock().clear();\n    }\n","sourceCodeStart":242,"sourceCodeEnd":278,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/hyperliquid/src/data.rs#L242-L278","documentation":"During teardown, await_pending_tasks asks the Hyperliquid data client's pending task set to shut down with bounded grace (1s) and drain (2s) timeouts. If tasks do not terminate within those bounds, finish_shutdown returns an error which is wrapped as 'Failed to terminate Hyperliquid data tasks'. This indicates stuck background data tasks at disconnect time.","triggerScenarios":"Calling teardown_partial_connect -> await_pending_tasks while a pending task (e.g. an in-flight HTTP/WS request loop) hangs and does not finish within the 1s begin/finish shutdown grace plus 2s drain window.","commonSituations":"Network stalls blocking in-flight requests during disconnect; task loops not observing the shutdown signal promptly; slow exchange responses at teardown time; deadlocks or very long awaits inside spawned tasks.","solutions":["Investigate which pending task hangs (enable task logging) and ensure it respects the shutdown signal and cancellation tokens","Add timeouts to individual in-flight requests so they cannot block teardown indefinitely","Increase finish_shutdown grace/drain durations if tasks legitimately need longer to wind down","Retry disconnect; if persistent, restart the client/node since a stuck task indicates a bug"],"exampleFix":"// before\n.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n// after\n.finish_shutdown(Duration::from_secs(5), Duration::from_secs(10)) // or fix the hanging task to honor shutdown","handlingStrategy":"try-catch","validationCode":"// best-effort pre-check: ensure no task is mid-request before teardown\nif !pending_tasks.is_empty() {\n    tracing::warn!(\"tearing down with {} pending tasks\", pending_tasks.len());\n}","typeGuard":"fn shutdown_within(d: Duration) -> impl Future<Output=bool> {\n    async move {\n        tokio::time::timeout(d, tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)))\n            .await.is_ok()\n    }\n}","tryCatchPattern":"if let Err(e) = await_pending_tasks().await {\n    tracing::warn!(\"{e}; forcing abort of pending tasks\");\n    pending_tasks.abort_all();\n}","preventionTips":["Ensure all spawned tasks honor shutdown signals/cancellation tokens","Add per-request timeouts so tasks cannot hang during teardown","Use generous drain durations for slow networks and log which task hangs"],"tags":["rust","hyperliquid","shutdown","async","timeout"],"backgroundTag":"request-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}