nautechsystems/nautilus_trader · error · anyhow::Error

Startup reconciliation timeout reached while requesting mass

Error message

Startup reconciliation timeout reached while requesting mass status from {client_id}

What it means

During startup reconciliation, the node requests an ExecutionMassStatus report from each execution client within config.timeout_reconciliation. If generate_mass_status for a given client_id does not complete before the remaining budget expires, the future is cancelled and this error is returned. It indicates the venue/adapter was too slow or hung producing its mass status report.

Source

Thrown at crates/live/src/node/mod.rs:888

                .checked_sub(elapsed)
                .expect("elapsed checked against reconciliation timeout");

            log_info!(
                "Requesting mass status from {}...",
                client_id,
                color = LogColor::Blue
            );

            let mass_status_result = dst::time::timeout(remaining, async {
                self.kernel
                    .exec_engine
                    .borrow_mut()
                    .generate_mass_status(&client_id, lookback_mins)
                    .await
            })
            .await
            .map_err(|_| {
                anyhow::anyhow!(
                    "Startup reconciliation timeout reached while requesting mass status from {client_id}"
                )
            })?;

            match mass_status_result {
                Ok(Some(mass_status)) => {
                    log_info!(
                        "Reconciling ExecutionMassStatus for {}",
                        client_id,
                        color = LogColor::Blue
                    );

                    let exec_engine_rc = self.kernel.exec_engine.clone();

                    let result = self
                        .exec_manager
                        .reconcile_execution_mass_status(mass_status, exec_engine_rc)
                        .await;

View on GitHub (pinned to 18893faf8b)

Solutions

1. Increase timeout_reconciliation in the live config to cover all clients' mass-status generation.
2. Reduce reconciliation_lookback_mins so the venue returns the report faster.
3. Check the venue/adapter API health and rate limits; add retry or backoff at the adapter level if supported.
4. Reduce the number of configured execution clients or stagger their startup so each gets sufficient timeout budget.

Example fix

// before
config.timeout_reconciliation = Duration::from_secs(10);
config.exec_engine.reconciliation_lookback_mins = Some(1440);
// after
config.timeout_reconciliation = Duration::from_secs(120);
config.exec_engine.reconciliation_lookback_mins = Some(60);

// Ensure reconciliation budget covers all clients
let est = clients.len() as u64 * per_client_report_secs;
assert!(config.timeout_reconciliation.as_secs() > est, "timeout_reconciliation too small for {clients:?}");

if let Err(e) = node.start().await {
    if e.to_string().contains("reconciliation timeout") {
        // raise timeout_reconciliation / lower lookback, then retry
    }
}

When it happens

Trigger: perform_startup_reconciliation loops over client ids; dst::time::timeout(remaining, generate_mass_status(&client_id, lookback_mins)) elapses (mod.rs:879-891), typically after earlier clients consumed most of timeout_reconciliation.

Common situations: Exchange REST endpoints rate-limiting or slow during high volatility; large reconciliation lookback (reconciliation_lookback_mins) making the report expensive; many execution clients sharing one timeout budget so later clients get almost no remaining time; unreliable API with long tail latencies.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• exec-connect timeout
• subscription confirmation failed: {e}
• Timed out requesting open orders for perm_id lookup
• {failure_prefix}; outcome is unknown after possible transmis
• Lighter WebSocket initial connection timeout after {} second

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/b930345fe47dddbf. Report an issue: GitHub.