nautechsystems/nautilus_trader · warning

Failed to terminate Derive data tasks: {e}

Error message

Failed to terminate Derive data tasks: {e}

What it means

Analogous to join_session_tasks, `join_pending_tasks` shuts down the Derive data client's pending (in-flight request) task group with 1s/2s grace periods. If the pending tasks do not finish within the deadlines, `finish_shutdown` errors and this message wraps it. It means outstanding data request tasks could not be cleanly awaited during disconnect.

Source

Thrown at crates/adapters/derive/src/data.rs:2231

    replay
}

impl DeriveDataClient {
    async fn join_session_tasks(&self) -> anyhow::Result<()> {
        self.session_tasks.begin_shutdown();
        self.session_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate Derive data session tasks: {e}"))?;
        Ok(())
    }

    async fn join_pending_tasks(&self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.pending_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate Derive data tasks: {e}"))?;
        Ok(())
    }
}

fn cache_instrument(
    instruments: &Arc<AtomicMap<InstrumentId, InstrumentAny>>,
    instrument: &InstrumentAny,
) {
    instruments.insert(instrument.id(), instrument.clone());
}

fn process_ticker_quote(
    msg: &DeriveTickerMsg,
    price_precision: u8,
    size_precision: u8,
    ts_init: UnixNanos,
    quote_cache: &mut QuoteCache,
) -> anyhow::Result<Option<QuoteTick>> {

View on GitHub (pinned to 18893faf8b)

Solutions

1. Retry shutdown once the network or in-flight requests settle.
2. Give in-flight requests shorter individual timeouts so pending tasks end promptly at shutdown.
3. Inspect the inner error to identify which pending task stalled.
4. Accept and log if this happens only at process exit and results are no longer needed.

match client.join_pending_tasks().await {
    Ok(()) => {}
    Err(e) => tracing::warn!("pending tasks did not shut down cleanly: {e:#}"),
}

When it happens

Trigger: Calling client disconnect/shutdown while request tasks (HTTP/ws request futures) are still pending and exceed the 1s/2s shutdown timeouts.

Common situations: Shutting down with many slow in-flight Derive REST requests; a hung HTTP call without a short timeout blocking the pending task group; abrupt teardown under load.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Failed to terminate Derive data session tasks: {e}
• Failed to terminate Derive execution tasks: {e}
• Hyperliquid WebSocket handler did not stop after abort
• Architect AX data WebSocket handler did not stop after abort
• Failed to finish Betfair data command tasks: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/ad6b3d8ca3176995. Report an issue: GitHub.