nautechsystems/nautilus_trader · critical · anyhow::Error

Reconnection timeout after {timeout_mins} minutes: {e}

Error message

Reconnection timeout after {timeout_mins} minutes: {e}

What it means

During live-feed reconnection, `run` tracks how long consecutive failed reconnect cycles have lasted. If `reconnect_timeout_mins` is configured and the elapsed time since `cycle_start` exceeds that budget, it emits an `Error` message with this text and breaks out with the last underlying error `e`. It prevents the client from retrying a dead connection indefinitely.

Source

Thrown at crates/adapters/databento/src/live.rs:499

                        log::info!("Resetting reconnection cycle after successful session");
                        reconnect_start = None;
                        attempt = 0;
                        self.backoff.reset();
                    } else {
                        log::debug!("Session ended normally");
                        break Ok(());
                    }
                }
                Err(e) => {
                    let cycle_start = reconnect_start.get_or_insert_with(tokio::time::Instant::now);

                    if let Some(timeout_mins) = self.reconnect_timeout_mins {
                        let elapsed = cycle_start.elapsed();
                        let timeout = Duration::from_mins(timeout_mins);

                        if elapsed >= timeout {
                            log::error!("Giving up reconnection after {timeout_mins} minutes");
                            self.send_msg(DatabentoMessage::Error(anyhow::anyhow!(
                                "Reconnection timeout after {timeout_mins} minutes: {e}"
                            )));
                            break Err(e);
                        }
                    }

                    let delay = self.backoff.next_duration();

                    log::warn!(
                        "Connection lost (attempt {}): {}. Reconnecting in {}s...",
                        attempt,
                        e,
                        delay.as_secs()
                    );

                    let sleep = tokio::time::sleep(delay);
                    tokio::pin!(sleep);

View on GitHub (pinned to 18893faf8b)

Solutions

1. Fix the underlying cause reported in the inner error `e` (connectivity, credentials, dataset)
2. Verify network access to Databento's live gateway and that the API key is valid
3. Increase `reconnect_timeout_mins` if transient outages legitimately last longer than the budget
4. Implement an outer supervisor that rebuilds and restarts the feed client after this terminal error
5. Alert/monitor on the emitted `DatabentoMessage::Error` so operators react before data loss grows

Example fix

// before
let client = DatabentoLiveClient::builder()
    .reconnect_timeout_mins(5)
    .build()?;

// after
let client = DatabentoLiveClient::builder()
    .reconnect_timeout_mins(30) // tolerate longer outages
    .build()?;

// before running: probe connectivity/credentials
let client = HttpClient::new();
client.get("https://hist.databento.com/v0/").await?; // fail fast if unreachable

match feed.run().await {
    Err(e) if e.to_string().contains("Reconnection timeout") => {
        log::error!("live feed gave up reconnecting: {e:#}");
        // page operator, then rebuild client and restart after backoff
    }
    other => other?,
}

When it happens

Trigger: The Databento live gateway is unreachable (network outage, invalid credentials causing immediate rejections, dataset/ gateway problems) so every reconnect cycle fails continuously for longer than `reconnect_timeout_mins` minutes.

Common situations: Network/firewall blocking `databento.com` live endpoints; expired API key causing endless auth rejections; Databento maintenance windows exceeding the timeout; a too-small `reconnect_timeout_mins` giving up during a transient outage.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• reconnection timed out after {}s
• reconnection timed out after {}s
• Failed to connect to Databento LSG: {e}
• Session ended by gateway
• Connection error: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/c507474ad6f96ebc. Report an issue: GitHub.