nautechsystems/nautilus_trader · warning

heartbeat request timeout should fit in an instant

Error message

heartbeat request timeout should fit in an instant

What it means

In the Polymarket heartbeat loop, each iteration computes a request deadline as current instant + HEARTBEAT_REQUEST_TIMEOUT via checked_add. The expect fires if the addition overflows tokio's Instant range, which normally cannot happen unless the process clock jumps far into the future or the timeout constant is astronomically large.

Source

Thrown at crates/adapters/polymarket/src/execution/lifecycle.rs:756

    let mut interval = tokio::time::interval(HEARTBEAT_INTERVAL);
    interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
    let mut heartbeat_id = String::new();
    let mut request_failures = 0;
    let mut last_acknowledged = None;

    loop {
        tokio::select! {
            () = cancellation.cancelled() => break,
            _ = interval.tick() => {}
        }

        let mut resynchronized = false;

        loop {
            let now = tokio::time::Instant::now();
            let request_timeout = now
                .checked_add(HEARTBEAT_REQUEST_TIMEOUT)
                .expect("heartbeat request timeout should fit in an instant");
            let health_deadline = last_acknowledged.map(|acknowledged: tokio::time::Instant| {
                acknowledged
                    .checked_add(heartbeat_health_timeout)
                    .expect("heartbeat health timeout should fit in an instant")
            });

            if health_deadline.is_some_and(|deadline| deadline <= now) {
                log::error!("Polymarket heartbeat health deadline elapsed");
                healthy.store(false, Ordering::Release);
                return;
            }

            let request_deadline =
                health_deadline.map_or(request_timeout, |deadline| deadline.min(request_timeout));
            let response = tokio::select! {
                () = cancellation.cancelled() => return,
                response = tokio::time::timeout_at(
                    request_deadline,

View on GitHub (pinned to 18893faf8b)

Solutions

1. Restart the process after the host clock has stabilized.
2. Fix the host's timekeeping (disable aggressive clock stepping, ensure stable TSC/KVM clock).
3. If patching, clamp the deadline instead of panicking: use min(now + timeout, Instant far-future bound).

Example fix

// before
let request_timeout = now
    .checked_add(HEARTBEAT_REQUEST_TIMEOUT)
    .expect("heartbeat request timeout should fit in an instant");

// after (defensive)
let request_timeout = now
    .checked_add(HEARTBEAT_REQUEST_TIMEOUT)
    .unwrap_or_else(|| tokio::time::Instant::now() + HEARTBEAT_REQUEST_TIMEOUT);

// supervise the heartbeat task and restart it if it dies
if let Err(e) = heartbeat_task.await {
    log::warn!("heartbeat task crashed ({e:?}); restarting");
}

When it happens

Trigger: run_heartbeats computing now + HEARTBEAT_REQUEST_TIMEOUT when the monotonic base instant is near its maximum — caused by extreme clock adjustments after VM pause/resume or corrupted monotonic time sources.

Common situations: Long-lived processes on hosts with unstable clock sources; VM/container migrations that reset monotonic time; nightly CI boxes resuming from suspend.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• heartbeat health margin should be shorter than the safety ti
• DurationNanos overflow in from_micros
• {e}
• Cannot increment time beyond u64::MAX
• MBO delta index overflow

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/01b1f0d471c70bbf. Report an issue: GitHub.