nautechsystems/nautilus_trader · error

Failed to terminate Derive execution tasks: {e}

Error message

Failed to terminate Derive execution tasks: {e}

What it means

During shutdown, await_pending_tasks asks the pending-tasks TaskGroup to finish_shutdown with 1s/2s grace timeouts. If the group cannot terminate its tasks within those windows (or is otherwise in a bad state), the error is wrapped and re-raised as this message. It signals that background execution tasks did not drain cleanly when the Derive execution client stopped.

Source

Thrown at crates/adapters/derive/src/execution.rs:332

            log::warn!("Skipping Derive {description} after shutdown began: {e}");
        }
    }

    fn abort_pending_tasks(&self) {
        self.pending_tasks.begin_shutdown();
    }

    fn abort_session_tasks(&self) {
        self.session_tasks.begin_shutdown();
        self.ws_client.begin_shutdown();
    }

    async fn await_pending_tasks(&self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.pending_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate Derive execution tasks: {e}"))?;
        Ok(())
    }

    async fn await_session_tasks(&self) -> anyhow::Result<()> {
        self.session_tasks.begin_shutdown();
        self.session_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| {
                anyhow::anyhow!("Failed to terminate Derive execution session tasks: {e}")
            })?;
        Ok(())
    }

    async fn ensure_instruments_initialized(&self) -> anyhow::Result<()> {
        if self.core.instruments_initialized() {
            return Ok(());
        }

View on GitHub (pinned to 18893faf8b)

Solutions

1. Retry shutdown once tasks have settled; transient overruns of the 1s/2s windows often succeed on a second attempt.
2. Check why pending tasks ignore cancellation (e.g. awaits without select on the cancellation token) and make them cooperative.
3. Increase availability headroom / fix network stalls so tasks complete before shutdown is requested.
4. Inspect the inner TaskGroup error in the message for the actual blocking cause.

// Before shutdown, check no tasks are pending:
if client.has_pending_work() { /* drain or wait before shutdown */ }

match client.await_pending_tasks().await {
    Ok(()) => (),
    Err(e) => {
        log::warn!("pending tasks did not drain cleanly: {e}; retrying");
        tokio::time::sleep(Duration::from_secs(2)).await;
        // retry or proceed with forced teardown
    }
}

When it happens

Trigger: Calling disconnect/shutdown on the Derive execution client while pending (in-flight) tasks are still running and do not finish within the 1s/2s shutdown grace periods; a hung task blocking TaskGroup::finish_shutdown.

Common situations: Slow or stalled WebSocket/network calls during teardown; long-running request futures that ignore the cancellation token; stopping a node or restarting the client while requests are in flight.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Failed to terminate Derive data session tasks: {e}
• Failed to terminate Derive data tasks: {e}
• Failed to terminate Derive execution session tasks: {e}
• Hyperliquid WebSocket handler did not stop after abort
• Architect AX data WebSocket handler did not stop after abort

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/ff2a89d43c70fed0. Report an issue: GitHub.