{"record":{"id":"73301c30c1f8f719","repo":"nautechsystems/nautilus_trader","slug":"failed-to-terminate-blockchain-submissions-e","errorCode":null,"errorMessage":"Failed to terminate blockchain submissions: {e}","messagePattern":"Failed to terminate blockchain submissions: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/execution/client.rs","lineNumber":5848,"sourceCode":"    }\n\n    async fn connect(&mut self) -> anyhow::Result<()> {\n        if self.core.is_connected() {\n            log::warn!(\"Blockchain execution client already connected\");\n            return Ok(());\n        }\n\n        log::info!(\n            \"Connecting to blockchain execution client on chain {}\",\n            self.chain.name\n        );\n\n        if !self.pending_tasks.is_open() || !self.pending_tasks.is_empty() {\n            self.pending_tasks.begin_shutdown();\n            self.pending_tasks\n                .finish_shutdown(Duration::from_secs(5), Duration::from_secs(2))\n                .await\n                .map_err(|e| anyhow::anyhow!(\"Failed to terminate blockchain submissions: {e}\"))?;\n            self.signer = None;\n            self.pending_tasks\n                .start_generation()\n                .map_err(|e| anyhow::anyhow!(\"Failed to start blockchain task generation: {e}\"))?;\n        }\n        release_preparing_slot(&self.in_flight);\n\n        let setup_guard = TaskGroupGuard::new(&[&self.pending_tasks], || {});\n\n        let payload_keys = PayloadKeySet::load(\n            self.config.payload_key_env.as_deref(),\n            &self.config.payload_key_retired_env,\n            self.config.payload_deployment_id.as_deref(),\n        )?\n        .map(Arc::new);\n\n        if self.cache.database.is_some() || self.config.postgres_cache_database_config.is_some() {\n            let keys = payload_keys.as_deref().ok_or_else(|| {","sourceCodeStart":5830,"sourceCodeEnd":5866,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/execution/client.rs#L5830-L5866","documentation":"During disconnect/stop, the client shuts down the pending_tasks queue that drains blockchain transaction submissions. finish_shutdown waits for in-flight submissions with bounded timeouts; if draining/terminating the task queue fails, the error is wrapped as \"Failed to terminate blockchain submissions\". This prevents silent loss or duplication of in-flight transactions during shutdown.","triggerScenarios":"Calling stop/disconnect while submissions are pending and the task queue fails to finish shutdown within its timeouts (finish_shutdown 5s soft / 2s hard), e.g. tasks blocked on a hung RPC or signer lock.","commonSituations":"Ethereum/JSON-RPC endpoint unresponsive at shutdown; a submission task awaiting confirmation stalls past the timeout; lock contention on the signer; slow network causing tasks to exceed the drain windows.","solutions":["Inspect the underlying source error in {e} to see whether tasks timed out or panicked, and fix the root cause (RPC health, signer lock)","Ensure the RPC endpoint is reachable and responsive before shutting down, or cancel stalled submissions first","Retry the disconnect after the pending task completes, or increase tolerance by draining submissions explicitly before stop","Restart the client cleanly if the task queue is wedged; verify no duplicate submissions were sent before resuming"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// before stopping, check no submissions are stuck\nif !client.pending_tasks_is_drained() {\n    log::warn!(\"submissions still pending; allow drain time before disconnect\");\n}","typeGuard":null,"tryCatchPattern":"match client.stop().await {\n    Err(e) if e.to_string().contains(\"Failed to terminate blockchain submissions\") => {\n        log::error!(\"shutdown drain failed: {e}; inspect RPC health and retry stop\");\n        tokio::time::sleep(Duration::from_secs(5)).await;\n        client.stop().await // retry once after drain window\n    }\n    other => other,\n}","preventionTips":["Keep the RPC endpoint healthy; monitor latency so submissions finish well within the 5s/2s drain windows","Avoid stopping the client while transactions are pending confirmation","Avoid shared signer locks that can stall submission tasks past the timeout","On failure, check for duplicate submissions before resuming"],"tags":["shutdown","async-tasks","timeout","submissions"],"backgroundTag":"request-timeout","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}