What causes "Failed to terminate Databento tasks: {e}" in nautechsystems/nautilus_trader?

Increase the shutdown timeout durations before reconnecting

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to terminate Databento tasks: {e}

Error message

Failed to terminate Databento tasks: {e}

What it means

During `connect`, any leftover task generation from a previous session must be shut down via `task_handles.finish_shutdown` before a new generation starts. If terminating the old tasks fails (tasks not finishing within the given timeouts), connect aborts with this error.

Source

Thrown at crates/adapters/databento/src/data.rs:500

        self.clear_feed_channels();
        self.abort_active_tasks();
        self.is_connected.store(false, Ordering::Relaxed);
        Ok(())
    }

    fn dispose(&mut self) -> anyhow::Result<()> {
        log::debug!("Disposing");
        self.stop()
    }

    async fn connect(&mut self) -> anyhow::Result<()> {
        log::debug!("Connecting...");

        if !self.task_handles.is_open() {
            self.task_handles
                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
                .await
                .map_err(|e| anyhow::anyhow!("Failed to terminate Databento tasks: {e}"))?;
            self.task_handles
                .start_generation()
                .map_err(|e| anyhow::anyhow!("Failed to start Databento task generation: {e}"))?;
            self.cancellation_token = self.task_handles.cancellation_token();
        }

        self.is_connected.store(true, Ordering::Relaxed);

        log::info!("Connected");
        Ok(())
    }

    async fn disconnect(&mut self) -> anyhow::Result<()> {
        log::debug!("Disconnecting...");

        self.send_close_to_active_feeds();
        self.clear_feed_channels();
        self.task_handles.begin_shutdown();

View on GitHub (pinned to 18893faf8b)

Solutions

Increase the shutdown timeout durations before reconnecting
Ensure disconnect() is called before a subsequent connect() so shutdown starts cleanly
Investigate why old tasks hang (e.g. blocked network reads) and add cancellation points
Retry connect() after a short delay once tasks have drained

Example fix

// before
client.connect().await?; // immediately after a previous session
// after
client.disconnect().await?;
tokio::time::sleep(Duration::from_secs(1)).await;
client.connect().await?;

Defensive patterns

Strategy: retry

Validate before calling

async fn clean_reconnect(client: &DatabentoDataClient) -> anyhow::Result<()> {
    client.disconnect().await.ok();
    tokio::time::sleep(Duration::from_millis(500)).await;
    client.connect().await
}

Try / catch

for attempt in 1..=3 {
    match client.connect().await {
        Ok(()) => break,
        Err(e) if e.to_string().contains("Failed to terminate Databento tasks") && attempt < 3 => {
            tokio::time::sleep(Duration::from_secs(attempt)).await;
        }
        Err(e) => return Err(e),
    }
}

Prevention

Call disconnect() before reconnecting to begin shutdown early
Serialize connect()/disconnect() calls; never run them concurrently
Allow cool-down time between reconnect cycles so old tasks can drain

When it happens

Trigger: Calling connect() while prior Databento tasks are still alive and fail to terminate within 1s graceful / 2s total shutdown windows.

Common situations: Rapid reconnect cycles where tasks are blocked on slow network I/O, wedged live-stream tasks stuck in a read, or calling connect() concurrently from two callers.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/d1ef01c31d56b329. Report an issue: GitHub.