What causes "Failed to calculate duration: {e}" in nautechsystems/nautilus_trader?

Retry the load once the system clock is synchronized (systemd-timesyncd/chronyd) — the cache data itself is unaffected.

Failed to calculate duration: {e}

Error message

Failed to calculate duration: {e}

What it means

At the end of load_cache, the engine logs how long the cache load took using SystemTime::now().duration_since(ts). If the system clock moved backwards since ts (manual adjustment, NTP correction, VM suspend/resume), duration_since returns an Err and the engine converts it into this anyhow error, aborting load_cache even though the cache itself loaded successfully.

Source

Thrown at crates/execution/src/engine/mod.rs:993

                    .map(|o| (o.instrument_id(), o.to_own_book_order()))
                    .collect()
            } else {
                Vec::new()
            }
        };

        for (instrument_id, own_order) in own_book_entries {
            let mut own_book = self.get_or_init_own_order_book(&instrument_id);
            own_book.add(own_order);
        }

        self.set_position_id_counts();

        log::info!(
            "Loaded cache in {}ms",
            SystemTime::now() // dst-ok: init-time log timing, not on DST state path
                .duration_since(ts)
                .map_err(|e| anyhow::anyhow!("Failed to calculate duration: {e}"))?
                .as_millis()
        );

        Ok(())
    }

    /// Flushes the database to persist all cached data.
    pub fn flush_db(&self) {
        self.cache.borrow_mut().flush_db();
    }

    /// Reconciles an execution report.
    pub fn reconcile_execution_report(&mut self, report: &ExecutionReport) {
        if !matches!(report, ExecutionReport::MassStatus(_)) {
            self.report_count += 1;
        }

        match report {

View on GitHub (pinned to 18893faf8b)

Solutions

Retry the load once the system clock is synchronized (systemd-timesyncd/chronyd) — the cache data itself is unaffected.
Replace the timing code with Instant (monotonic) instead of SystemTime for elapsed-time measurement.
Downgrade the duration calculation to a non-fatal log path so a clock jump doesn't fail cache loading.
Ensure the host clock is synced (NTP enabled) before starting the node.

Example fix

// before
let elapsed = SystemTime::now()
    .duration_since(ts)
    .map_err(|e| anyhow::anyhow!("Failed to calculate duration: {e}"))?;
// after
let elapsed = ts.elapsed(); // ts: Instant — monotonic, cannot go backwards

Defensive patterns

Strategy: try-catch

Try / catch

// tolerate clock jumps during init-time logging
match SystemTime::now().duration_since(ts) {
    Ok(d) => log::info!("Loaded cache in {}ms", d.as_millis()),
    Err(_) => log::warn!("clock stepped backwards during cache load"),
}

Prevention

Enable NTP/chrony so the host clock does not step backwards at runtime.
Prefer Instant over SystemTime for measuring elapsed durations.
Avoid running nodes on hosts that suspend/resume or share unmanaged clocks (VMs).

When it happens

Trigger: Calling load_cache (e.g. during execution engine initialization) when SystemTime::now() is earlier than the timestamp ts captured at the start of the load — the OS clock was set backwards during the load.

Common situations: NTP stepping the clock backward on a VM or container host during startup; laptop resumed from sleep with a stale clock; operator manually changing system time; running in a VM with paused/resumed host time.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/abae70509ac3565b. Report an issue: GitHub.