nautechsystems/nautilus_trader · error

Failed to terminate Derive execution session tasks: {e}

Error message

Failed to terminate Derive execution session tasks: {e}

What it means

Analogous to await_pending_tasks, await_session_tasks tries to drain the session-task group (WebSocket dispatch/session lifecycle tasks) with 1s/2s grace timeouts. If those session tasks cannot finish in time or the group reports an error, it is wrapped as this message. It means the Derive execution client's session tasks failed to terminate cleanly at shutdown.

Source

Thrown at crates/adapters/derive/src/execution.rs:342

        self.ws_client.begin_shutdown();
    }

    async fn await_pending_tasks(&self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.pending_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate Derive execution tasks: {e}"))?;
        Ok(())
    }

    async fn await_session_tasks(&self) -> anyhow::Result<()> {
        self.session_tasks.begin_shutdown();
        self.session_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| {
                anyhow::anyhow!("Failed to terminate Derive execution session tasks: {e}")
            })?;
        Ok(())
    }

    async fn ensure_instruments_initialized(&self) -> anyhow::Result<()> {
        if self.core.instruments_initialized() {
            return Ok(());
        }
        // Lazy bootstrap: exec-side fetches per-instrument on first reference.
        // Marking the flag prevents duplicate work across reconnect cycles.
        self.core.set_instruments_initialized();
        Ok(())
    }

    fn reconciliation_context(&self) -> DeriveReconciliationContext {
        DeriveReconciliationContext {
            http_client: self.http_client.clone(),
            emitter: self.emitter.clone(),

View on GitHub (pinned to 18893faf8b)

Solutions

1. Ensure the WS connection is closed/healthy before shutdown so the dispatch loop exits promptly.
2. Retry shutdown; if persistent, inspect why the session task doesn't honor the cancellation token.
3. Fix network stalls or make the WS read use tokio::select! against the cancellation token.
4. Read the wrapped inner error for the TaskGroup's specific failure reason.

// Ensure the WS connection is closed before shutdown:
if client.is_ws_connected() { client.disconnect_ws().await?; }

if let Err(e) = client.await_session_tasks().await {
    log::error!("Derive session tasks failed to terminate: {e}");
    // proceed with process exit or force-abort the task group
}

When it happens

Trigger: Disconnecting the Derive execution client while the WebSocket dispatch loop/session tasks are still active and non-responsive past the grace timeouts; finish_shutdown returning an error for the session_tasks group.

Common situations: WS connection in a bad state (stuck read) during teardown; process shutdown while a reconnect loop is mid-flight; blocking code inside the session task loop preventing timely cancellation.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Failed to terminate Derive execution tasks: {e}
• joined shutdown errors (std::mem::take(&mut self.shutdown_er
• Hyperliquid WebSocket handler did not stop after abort
• Failed to terminate AX data tasks: {e}
• Failed to terminate AX data session tasks: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/093aa9723d161490. Report an issue: GitHub.