What causes "Failed to terminate blockchain submissions: {e}" in nautechsystems/nautilus_trader?

Inspect the underlying source error in {e} to see whether tasks timed out or panicked, and fix the root cause (RPC health, signer lock)

nautechsystems/nautilus_trader · error

Failed to terminate blockchain submissions: {e}

Error message

Failed to terminate blockchain submissions: {e}

What it means

During disconnect/stop, the client shuts down the pending_tasks queue that drains blockchain transaction submissions. finish_shutdown waits for in-flight submissions with bounded timeouts; if draining/terminating the task queue fails, the error is wrapped as "Failed to terminate blockchain submissions". This prevents silent loss or duplication of in-flight transactions during shutdown.

Source

Thrown at crates/adapters/blockchain/src/execution/client.rs:5848

    }

    async fn connect(&mut self) -> anyhow::Result<()> {
        if self.core.is_connected() {
            log::warn!("Blockchain execution client already connected");
            return Ok(());
        }

        log::info!(
            "Connecting to blockchain execution client on chain {}",
            self.chain.name
        );

        if !self.pending_tasks.is_open() || !self.pending_tasks.is_empty() {
            self.pending_tasks.begin_shutdown();
            self.pending_tasks
                .finish_shutdown(Duration::from_secs(5), Duration::from_secs(2))
                .await
                .map_err(|e| anyhow::anyhow!("Failed to terminate blockchain submissions: {e}"))?;
            self.signer = None;
            self.pending_tasks
                .start_generation()
                .map_err(|e| anyhow::anyhow!("Failed to start blockchain task generation: {e}"))?;
        }
        release_preparing_slot(&self.in_flight);

        let setup_guard = TaskGroupGuard::new(&[&self.pending_tasks], || {});

        let payload_keys = PayloadKeySet::load(
            self.config.payload_key_env.as_deref(),
            &self.config.payload_key_retired_env,
            self.config.payload_deployment_id.as_deref(),
        )?
        .map(Arc::new);

        if self.cache.database.is_some() || self.config.postgres_cache_database_config.is_some() {
            let keys = payload_keys.as_deref().ok_or_else(|| {

View on GitHub (pinned to 18893faf8b)

Solutions

Inspect the underlying source error in {e} to see whether tasks timed out or panicked, and fix the root cause (RPC health, signer lock)
Ensure the RPC endpoint is reachable and responsive before shutting down, or cancel stalled submissions first
Retry the disconnect after the pending task completes, or increase tolerance by draining submissions explicitly before stop
Restart the client cleanly if the task queue is wedged; verify no duplicate submissions were sent before resuming

Defensive patterns

Strategy: try-catch

Validate before calling

// before stopping, check no submissions are stuck
if !client.pending_tasks_is_drained() {
    log::warn!("submissions still pending; allow drain time before disconnect");
}

Try / catch

match client.stop().await {
    Err(e) if e.to_string().contains("Failed to terminate blockchain submissions") => {
        log::error!("shutdown drain failed: {e}; inspect RPC health and retry stop");
        tokio::time::sleep(Duration::from_secs(5)).await;
        client.stop().await // retry once after drain window
    }
    other => other,
}

Prevention

Keep the RPC endpoint healthy; monitor latency so submissions finish well within the 5s/2s drain windows
Avoid stopping the client while transactions are pending confirmation
Avoid shared signer locks that can stall submission tasks past the timeout
On failure, check for duplicate submissions before resuming

When it happens

Trigger: Calling stop/disconnect while submissions are pending and the task queue fails to finish shutdown within its timeouts (finish_shutdown 5s soft / 2s hard), e.g. tasks blocked on a hung RPC or signer lock.

Common situations: Ethereum/JSON-RPC endpoint unresponsive at shutdown; a submission task awaiting confirmation stalls past the timeout; lock contention on the signer; slow network causing tasks to exceed the drain windows.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/73301c30c1f8f719. Report an issue: GitHub.