nautechsystems/nautilus_trader · error

Failed to terminate Tardis tasks

Error message

Failed to terminate Tardis tasks: {e}

What it means

Raised in connect when finish_shutdown on the Tardis task set fails (does not complete within the 1s/2s grace windows) while preparing for a (re)connection. The prior generation of stream tasks could not be terminated cleanly, so connect aborts.

Solutions

  1. Ensure the WS task loop selects on the cancellation token so it exits promptly.
  2. Increase the finish_shutdown grace durations if tasks legitimately need longer to stop.
  3. Check network conditions toward the Tardis Machine; a hung socket can stall graceful shutdown.
  4. Drop/recreate the client if the task set is wedged.

Example fix

// before
.map_err(|e| anyhow::anyhow!("Failed to terminate Tardis tasks: {e}"))?;
// after: make the ws loop cancellation-responsive
// tokio::select! { _ = cancellation_token.cancelled() => break, msg = ws.next() => ... }
Defensive patterns

Strategy: retry

Try / catch

match connect().await {
    Err(e) if e.to_string().contains("Failed to terminate Tardis tasks") => {
        log::warn!("previous tasks wedged; restarting client");
        client = TardisDataClient::new(config)?; // recreate instead of retrying in-place
        client.connect().await?;
    }
    other => other?,
}

Prevention

When it happens

Trigger: Calling connect while existing WebSocket tasks ignore the cancellation token and keep running past the finish_shutdown timeouts.

Common situations: WebSocket read loop blocked on a stalled Tardis Machine connection without honoring cancellation; very slow network teardown; tasks awaiting a long I/O future between cancellation checks.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors


AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/6ba8bf3c79c486b4. Report an issue: GitHub.

Appendix: source

Thrown at crates/adapters/tardis/src/data.rs:518

    fn unsubscribe_funding_rates(&mut self, _cmd: &UnsubscribeFundingRates) -> anyhow::Result<()> {
        Ok(())
    }

    async fn connect(&mut self) -> anyhow::Result<()> {
        if self.is_connected() && self.tasks.is_open() {
            return Ok(());
        }

        if self.config.options.is_empty() && self.config.stream_options.is_empty() {
            anyhow::bail!("Either replay `options` or `stream_options` must be provided");
        }

        if !self.tasks.is_open() {
            self.tasks
                .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
                .await
                .map_err(|e| anyhow::anyhow!("Failed to terminate Tardis tasks: {e}"))?;
            self.tasks
                .start_generation()
                .map_err(|e| anyhow::anyhow!("Failed to start Tardis task generation: {e}"))?;
            self.cancellation_token = self.tasks.cancellation_token();
        }

        let is_stream_mode = self.is_stream_mode();
        let book_snapshot_output = self.config.book_snapshot_output.clone();
        let extract_bbo_as_quotes = self.config.extract_bbo_as_quotes;

        let http_client = TardisHttpClient::new(
            self.config
                .api_key
                .as_ref()
                .map(|value| value.expose_secret()),
            None,
            None,
            self.config.normalize_symbols,

View on GitHub (pinned to 18893faf8b)