nautechsystems/nautilus_trader · error · anyhow::Error

Failed to terminate Databento tasks

Error message

Failed to terminate Databento tasks: {e}

What it means

During `connect`, any leftover task generation from a previous session must be shut down via `task_handles.finish_shutdown` before a new generation starts. If terminating the old tasks fails (tasks not finishing within the given timeouts), connect aborts with this error.

Solutions

  1. Increase the shutdown timeout durations before reconnecting
  2. Ensure disconnect() is called before a subsequent connect() so shutdown starts cleanly
  3. Investigate why old tasks hang (e.g. blocked network reads) and add cancellation points
  4. Retry connect() after a short delay once tasks have drained

Example fix

// before
client.connect().await?; // immediately after a previous session
// after
client.disconnect().await?;
tokio::time::sleep(Duration::from_secs(1)).await;
client.connect().await?;
Defensive patterns

Strategy: retry

Validate before calling

async fn clean_reconnect(client: &DatabentoDataClient) -> anyhow::Result<()> {
    client.disconnect().await.ok();
    tokio::time::sleep(Duration::from_millis(500)).await;
    client.connect().await
}

Try / catch

for attempt in 1..=3 {
    match client.connect().await {
        Ok(()) => break,
        Err(e) if e.to_string().contains("Failed to terminate Databento tasks") && attempt < 3 => {
            tokio::time::sleep(Duration::from_secs(attempt)).await;
        }
        Err(e) => return Err(e),
    }
}

Prevention

When it happens

Trigger: Calling connect() while prior Databento tasks are still alive and fail to terminate within 1s graceful / 2s total shutdown windows.

Common situations: Rapid reconnect cycles where tasks are blocked on slow network I/O, wedged live-stream tasks stuck in a read, or calling connect() concurrently from two callers.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/d1ef01c31d56b329. Report an issue: GitHub.

Appendix: source

Thrown at crates/adapters/databento/src/data.rs:500

        self.clear_feed_channels();
        self.abort_active_tasks();
        self.is_connected.store(false, Ordering::Relaxed);
        Ok(())
    }

    fn dispose(&mut self) -> anyhow::Result<()> {
        log::debug!("Disposing");
        self.stop()
    }

    async fn connect(&mut self) -> anyhow::Result<()> {
        log::debug!("Connecting...");

        if !self.task_handles.is_open() {
            self.task_handles
                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
                .await
                .map_err(|e| anyhow::anyhow!("Failed to terminate Databento tasks: {e}"))?;
            self.task_handles
                .start_generation()
                .map_err(|e| anyhow::anyhow!("Failed to start Databento task generation: {e}"))?;
            self.cancellation_token = self.task_handles.cancellation_token();
        }

        self.is_connected.store(true, Ordering::Relaxed);

        log::info!("Connected");
        Ok(())
    }

    async fn disconnect(&mut self) -> anyhow::Result<()> {
        log::debug!("Disconnecting...");

        self.send_close_to_active_feeds();
        self.clear_feed_channels();
        self.task_handles.begin_shutdown();

View on GitHub (pinned to 18893faf8b)