{"record":{"id":"391a9f3a570b090f","repo":"nautechsystems/nautilus_trader","slug":"failed-to-terminate-hyperliquid-execution-tasks","errorCode":null,"errorMessage":"Failed to terminate Hyperliquid execution tasks: {e}","messagePattern":"Failed to terminate Hyperliquid execution tasks: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/hyperliquid/src/execution.rs","lineNumber":644,"sourceCode":"        }\n\n        if let Err(e) = self.await_pending_tasks().await {\n            self.shutdown_errors.push(e.to_string());\n        }\n        self.core.set_disconnected();\n\n        if !self.shutdown_errors.is_empty() {\n            anyhow::bail!(std::mem::take(&mut self.shutdown_errors).join(\"; \"));\n        }\n        Ok(())\n    }\n\n    async fn await_pending_tasks(&self) -> anyhow::Result<()> {\n        self.pending_tasks.begin_shutdown();\n        self.pending_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to terminate Hyperliquid execution tasks: {e}\"))?;\n        Ok(())\n    }\n\n    async fn await_session_tasks(&self) -> anyhow::Result<()> {\n        self.session_tasks.begin_shutdown();\n        self.session_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| {\n                anyhow::anyhow!(\"Failed to terminate Hyperliquid execution session tasks: {e}\")\n            })?;\n        Ok(())\n    }\n}\n\n#[async_trait(?Send)]\nimpl ExecutionClient for HyperliquidExecutionClient {\n    fn is_connected(&self) -> bool {","sourceCodeStart":626,"sourceCodeEnd":662,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/hyperliquid/src/execution.rs#L626-L662","documentation":"This error is raised when the Hyperliquid execution client cannot cleanly terminate its pending-task group within the allotted shutdown timeouts during teardown. The TaskGroup's finish_shutdown gives tasks a grace period (1s) plus a hard cancel window (2s) before giving up and returning an error, which is then wrapped with this message.","triggerScenarios":"await_pending_tasks() is called from teardown_partial_connect() when a connect attempt fails partway; it errors if any spawned pending tasks (in-flight order submissions/cancellations) are still running after begin_shutdown and fail to finish within the 1s/2s shutdown deadlines.","commonSituations":"A partially-completed connect where the WebSocket or HTTP side failed but spawned tasks are blocked on slow network I/O; stalled Hyperliquid API calls hanging past the 2s hard-cancel; event-loop congestion delaying task completion.","solutions":["Retry the teardown or client teardown after a short delay so blocked tasks finish or get dropped","Investigate why in-flight tasks hang (network latency, unresponsive Hyperliquid endpoint) and add request timeouts","Check logs from the underlying finish_shutdown error for the specific task that refused to terminate","Increase shutdown grace/cancel durations in the TaskGroup configuration if tasks legitimately need longer"],"exampleFix":"// before\nself.pending_tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)).await.map_err(...)?\n// after\n// on repeated failure, log and force-drop instead of failing teardown:\nif let Err(e) = self.pending_tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)).await {\n    log::warn!(\"Hyperliquid pending tasks did not terminate cleanly: {e}\");\n}","handlingStrategy":"try-catch","validationCode":"if !self.pending_tasks.is_open() { self.teardown_partial_connect().await?; }","typeGuard":null,"tryCatchPattern":"match client.await_pending_tasks().await { Err(e) => log::warn!(\"pending tasks did not terminate: {e}; retrying teardown\") => retry_teardown(), Ok(_) => {} }","preventionTips":["Add request timeouts to in-flight order tasks so they finish within shutdown windows","Monitor shutdown logs for repeated termination failures","Avoid rapid connect/disconnect cycles","Keep event loops unblocked so tasks can observe shutdown signals"],"tags":["rust","shutdown","task-group","hyperliquid"],"backgroundTag":"request-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}