nautechsystems/nautilus_trader · warning

Failed to terminate Derive data session tasks: {e}

Error message

Failed to terminate Derive data session tasks: {e}

What it means

During Derive data-client shutdown, `join_session_tasks` asks the session task group to begin and finish shutdown with fixed grace periods (1s begin / 2s finish). If `finish_shutdown` returns an error (tasks did not terminate within the timeouts), it is wrapped with this message and propagated as an anyhow error. It indicates websocket/session bookkeeping tasks for the Derive data client could not be cleanly joined.

Source

Thrown at crates/adapters/derive/src/data.rs:2222

                    | DeriveWsError::NotConnected
            )
        });

    if replay {
        ws.remember_subscription(channel);
    } else {
        ws.forget_subscription(channel);
    }
    replay
}

impl DeriveDataClient {
    async fn join_session_tasks(&self) -> anyhow::Result<()> {
        self.session_tasks.begin_shutdown();
        self.session_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate Derive data session tasks: {e}"))?;
        Ok(())
    }

    async fn join_pending_tasks(&self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.pending_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate Derive data tasks: {e}"))?;
        Ok(())
    }
}

fn cache_instrument(
    instruments: &Arc<AtomicMap<InstrumentId, InstrumentAny>>,
    instrument: &InstrumentAny,
) {
    instruments.insert(instrument.id(), instrument.clone());

View on GitHub (pinned to 18893faf8b)

Solutions

1. Retry shutdown; the error is often transient if the network recovers.
2. Check network/proxy health so session tasks can complete pending I/O and observe the shutdown signal.
3. Inspect the inner error `{e}` to see which tasks failed and whether they are stuck in a reconnect loop.
4. Log and continue if acceptable — this occurs during teardown, after which remaining tasks can be abandoned.

match client.join_session_tasks().await {
    Ok(()) => {}
    Err(e) => tracing::warn!("session tasks did not shut down cleanly: {e:#}"),
}

When it happens

Trigger: Calling client disconnect/shutdown while Derive data session tasks are still busy (e.g. blocked on network I/O or an unresponsive websocket) so they exceed the 1s/2s shutdown timeouts.

Common situations: Terminating the process with live Derive websocket subscriptions; a hung network connection preventing session tasks from observing the shutdown signal; slow event-loop shutdown under heavy load.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Failed to terminate Derive data tasks: {e}
• Failed to terminate Derive execution tasks: {e}
• Hyperliquid WebSocket handler did not stop after abort
• Architect AX data WebSocket handler did not stop after abort
• Failed to finish Betfair data command tasks: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/1bf4ba9da8590286. Report an issue: GitHub.