{"record":{"id":"ff2a89d43c70fed0","repo":"nautechsystems/nautilus_trader","slug":"failed-to-terminate-derive-execution-tasks-e","errorCode":null,"errorMessage":"Failed to terminate Derive execution tasks: {e}","messagePattern":"Failed to terminate Derive execution tasks: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/derive/src/execution.rs","lineNumber":332,"sourceCode":"            log::warn!(\"Skipping Derive {description} after shutdown began: {e}\");\n        }\n    }\n\n    fn abort_pending_tasks(&self) {\n        self.pending_tasks.begin_shutdown();\n    }\n\n    fn abort_session_tasks(&self) {\n        self.session_tasks.begin_shutdown();\n        self.ws_client.begin_shutdown();\n    }\n\n    async fn await_pending_tasks(&self) -> anyhow::Result<()> {\n        self.pending_tasks.begin_shutdown();\n        self.pending_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to terminate Derive execution tasks: {e}\"))?;\n        Ok(())\n    }\n\n    async fn await_session_tasks(&self) -> anyhow::Result<()> {\n        self.session_tasks.begin_shutdown();\n        self.session_tasks\n            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))\n            .await\n            .map_err(|e| {\n                anyhow::anyhow!(\"Failed to terminate Derive execution session tasks: {e}\")\n            })?;\n        Ok(())\n    }\n\n    async fn ensure_instruments_initialized(&self) -> anyhow::Result<()> {\n        if self.core.instruments_initialized() {\n            return Ok(());\n        }","sourceCodeStart":314,"sourceCodeEnd":350,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/derive/src/execution.rs#L314-L350","documentation":"During shutdown, await_pending_tasks asks the pending-tasks TaskGroup to finish_shutdown with 1s/2s grace timeouts. If the group cannot terminate its tasks within those windows (or is otherwise in a bad state), the error is wrapped and re-raised as this message. It signals that background execution tasks did not drain cleanly when the Derive execution client stopped.","triggerScenarios":"Calling disconnect/shutdown on the Derive execution client while pending (in-flight) tasks are still running and do not finish within the 1s/2s shutdown grace periods; a hung task blocking TaskGroup::finish_shutdown.","commonSituations":"Slow or stalled WebSocket/network calls during teardown; long-running request futures that ignore the cancellation token; stopping a node or restarting the client while requests are in flight.","solutions":["Retry shutdown once tasks have settled; transient overruns of the 1s/2s windows often succeed on a second attempt.","Check why pending tasks ignore cancellation (e.g. awaits without select on the cancellation token) and make them cooperative.","Increase availability headroom / fix network stalls so tasks complete before shutdown is requested.","Inspect the inner TaskGroup error in the message for the actual blocking cause."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before shutdown, check no tasks are pending:\nif client.has_pending_work() { /* drain or wait before shutdown */ }","typeGuard":null,"tryCatchPattern":"match client.await_pending_tasks().await {\n    Ok(()) => (),\n    Err(e) => {\n        log::warn!(\"pending tasks did not drain cleanly: {e}; retrying\");\n        tokio::time::sleep(Duration::from_secs(2)).await;\n        // retry or proceed with forced teardown\n    }\n}","preventionTips":["Make background tasks honor the cancellation token via tokio::select!","Quiesce in-flight requests before initiating shutdown","Treat shutdown-timeout errors as non-fatal if the process is exiting anyway"],"tags":["derive","shutdown","task-group","async","timeout"],"backgroundTag":"invalid-state-transition","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}