ruvnet/ruflo · error · HlcSkewError

HLC skew exceeded: received physicalMs=${receivedPhysicalMs}

Error message

HLC skew exceeded: received physicalMs=${receivedPhysicalMs} vs local=${localPhysicalMs} (max=${maxSkewMs}ms)

What it means

Hlc.update(received) rejects any remote timestamp whose physicalMs is more than maxSkewMs ahead of the local wall clock. The guard is deliberate: adopting a future timestamp would poison the hybrid logical clock's timeline for every participant, so it throws HlcSkewError and never jumps the local clock forward. The error message carries received vs local physicalMs plus the allowed skew so the offending node can be identified.

Source

Thrown at v3/@claude-flow/claims/src/infrastructure/hlc.ts:142

      logical = 0;
    } else {
      // Wall clock didn't advance (or went backward); keep last physical and bump logical.
      physicalMs = this.last.physicalMs;
      logical = this.last.logical + 1;
    }

    this.last = { physicalMs, logical, nodeId: this.nodeId };
    return this.last;
  }

  update(received: HlcTimestamp): HlcTimestamp {
    const wall = this.physicalClock();

    // Skew guard: refuse HLCs that are too far in the future.
    // We DO NOT jump local clock forward to match — a misbehaving peer would
    // poison the global timeline. Instead we throw and let the caller decide.
    if (received.physicalMs > wall + this.maxSkewMs) {
      throw new HlcSkewError(received.physicalMs, wall, this.maxSkewMs);
    }

    const maxPhysical = Math.max(wall, this.last.physicalMs, received.physicalMs);

    let logical: number;
    if (maxPhysical === this.last.physicalMs && maxPhysical === received.physicalMs) {
      logical = Math.max(this.last.logical, received.logical) + 1;
    } else if (maxPhysical === this.last.physicalMs) {
      logical = this.last.logical + 1;
    } else if (maxPhysical === received.physicalMs) {
      logical = received.logical + 1;
    } else {
      logical = 0;
    }

    this.last = { physicalMs: maxPhysical, logical, nodeId: this.nodeId };
    return this.last;
  }

View on GitHub (pinned to fa13ee4ad6)

Solutions

Sync clocks on every node: run chrony/ntpd/systemd-timesyncd and verify with timedatectl or chronyc tracking
Catch HlcSkewError around applyRemoteEvent, quarantine the offending event (or retry after clocks resync) instead of letting it kill the sync loop
If drift is legitimate and measured, raise maxSkewMs on the HLC configuration
Fix the source node identified by the received-vs-local values in the message; do not compensate by forwarding local time

Example fix

// before
await store.applyRemoteEvent(payload.event, payload.vclock, payload.hlc);

// after
import { HlcSkewError } from '../infrastructure/hlc.js';
try {
  await store.applyRemoteEvent(payload.event, payload.vclock, payload.hlc);
} catch (e) {
  if (e instanceof HlcSkewError) {
    deadLetter.push({ event: payload.event, reason: 'clock-skew', detail: e.message });
    return;
  }
  throw e;
}

Defensive patterns

Strategy: try-catch

Validate before calling

const drift = payload.hlc.physicalMs - Date.now();
if (drift > MAX_EXPECTED_SKEW_MS) {
  deadLetter.push({ event: payload.event, reason: 'future-dated hlc', driftMs: drift });
} else {
  await store.applyRemoteEvent(payload.event, payload.vclock, payload.hlc);
}

Try / catch

import { HlcSkewError } from './hlc.js';
try { await store.applyRemoteEvent(ev, vclock, hlc); }
catch (e) {
  if (e instanceof HlcSkewError) { deadLetter.push({ ev, reason: e.message }); return; }
  throw e;
}

Prevention

Run NTP/chrony on every federated node and alert on drift >1s
Never forward-jump the local HLC or wall clock to match a peer
Set maxSkewMs from measured worst-case drift, and quarantine (not crash) on violation
For event replay, strip old hlc timestamps or replay through a fresh clock path

When it happens

Trigger: applyRemoteEvent (or a direct hlc.update call) with a remote HLC from a node whose system clock runs fast: no NTP, a VM resumed from suspend, a container with skewed time, or replayed recorded events whose timestamps now sit far in the future relative to a reinitialized local clock; maxSkewMs configured smaller than real drift.

Common situations: Cloud VMs/containers without chrony/ntpd; laptop sleep/wake cycles during federated tests; CI runners with drifted clocks; cross-datacenter replication where one site's clock wanders.

Related errors

AI-assisted analysis of ruvnet/ruflo@fa13ee4ad6 (2026-08-18). Data as JSON: /api/errors/2a063e722421c478. Report an issue: GitHub.