nautechsystems/nautilus_trader · error

Failed to terminate {owner} tasks: {e}

Error message

Failed to terminate {owner} tasks: {e}

What it means

terminate_tasks asks the shared TaskGroup to shut down its spawned tasks within a graceful timeout, then an abort timeout. If finish_shutdown returns an error (tasks ignored cancellation or hung past the abort bound), it is wrapped as "Failed to terminate {owner} tasks". It indicates owned background tasks (instrument refresh, data tasks) did not stop cleanly during teardown/reset.

Source

Thrown at crates/adapters/okx/src/common/task.rs:50

        tokio::select! {
            biased;
            () = cancel.cancelled() => {}
            () = fut => {}
        }
    }) {
        log::debug!("Skipping task spawn after OKX shutdown began: {e}");
    }
}

/// Gracefully completes and then aborts every task retained by `tasks`.
///
/// The scope remains closed if any handle outlives the forced completion bound.
pub(crate) async fn terminate_tasks(tasks: &TaskGroup, owner: &str) -> anyhow::Result<()> {
    tasks.begin_shutdown();
    tasks
        .finish_shutdown(TASK_GRACEFUL_TIMEOUT, TASK_ABORT_TIMEOUT)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to terminate {owner} tasks: {e}"))?;
    Ok(())
}

#[cfg(test)]
mod tests {
    use std::sync::{
        Arc,
        atomic::{AtomicBool, Ordering},
    };

    use super::*;

    #[tokio::test]
    async fn termination_cancels_registered_task() {
        let tasks = TaskGroup::new();
        let spawner = tasks.spawner().expect("task spawner");
        let canceled = Arc::new(AtomicBool::new(false));
        let signal = DropSignal(Arc::clone(&canceled));

View on GitHub (pinned to 18893faf8b)

Solutions

1. Read the inner error from the message to see whether tasks timed out gracefully or at the abort bound.
2. Ensure spawned tasks honor cancellation (select on the shutdown signal / tokio cancellation tokens) rather than blocking awaits.
3. Increase TASK_GRACEFUL_TIMEOUT/TASK_ABORT_TIMEOUT only if tasks legitimately need longer (e.g. slow network); prefer making tasks cancellable.
4. Check for deadlocks or unbounded loops in the task body; the scope cannot close if a handle outlives the abort bound.
5. Log and continue on teardown in non-critical paths, since this occurs during shutdown anyway.

Example fix

// before (task body ignores shutdown)
loop {
    let snapshot = fetch_instruments().await?;
    publish(snapshot).await;
}
// after
loop {
    tokio::select! {
        _ = shutdown.cancelled() => break,
        snapshot = fetch_instruments() => publish(snapshot.await).await,
    }
}

match terminate_tasks(&task_group, "OkxExecutionClient").await {
    Ok(()) => {}
    Err(e) => tracing::error!("teardown: {e:#}"), // continue shutdown
}

When it happens

Trigger: Calling terminate_tasks during client stop/reset/teardown when a registered task is blocked (e.g. awaiting a network call that ignores the shutdown signal, a long poll, or a deadlock) and exceeds TASK_GRACEFUL_TIMEOUT then TASK_ABORT_TIMEOUT.

Common situations: Live OKX adapter shutdown while a WebSocket reconnect loop or HTTP instrument refresh is in flight; a stuck request to OKX during network outage; test teardown with a task that never yields.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Timed out joining WebSocket handler task after abort: {error
• Failed to finish Betfair data command tasks: {e}
• Failed to terminate Binance execution tasks: {e}
• Hyperliquid WebSocket handler did not stop after abort
• Lighter WebSocket handler did not stop after abort

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/ab4ec8b031359c15. Report an issue: GitHub.