koala73/worldmonitor · critical · Error

[runDailyRamp] inFlight bounds-check invariant violated

Error message

[runDailyRamp] inFlight bounds-check invariant violated

What it means

In runDailyRamp, after confirming inFlight.length > 0 the code reads inFlight[0] and defensively guards against it being undefined (silences noUncheckedIndexedAccess and protects against a future change breaking the length check). This throw is an invariant violation: it cannot fire under normal JS semantics if the length check is intact.

Source

Thrown at convex/broadcast/rampRunner.ts:1058

    // module so the operator-recovery commands (`recoverFromPartialFailure`,
    // `clearPartialFailure`, `forceReleaseLease`) keep working on any
    // legacy partial-failure rows. New runs go through the state machine.
    //
    // In-flight guard: refuse to start a new wave while any waveRuns row is
    // active. The new state machine maintains its own `pendingRunId` lease
    // on `broadcastRampConfig` via `_claimWaveRunLease` — this guard is
    // belt-and-suspenders against a force-cleared lease leaving an orphaned
    // active waveRuns row.
    const inFlight = await ctx.runQuery(
      internal.broadcast.waveRuns._listInFlightWaveRuns,
      {},
    );
    if (inFlight.length > 0) {
      const top = inFlight[0];
      // top is guaranteed non-null by the length check (silences
      // noUncheckedIndexedAccess and protects future code changes that
      // might break the bounds check).
      if (!top) throw new Error("[runDailyRamp] inFlight bounds-check invariant violated");
      const ageMin = (Date.now() - top.lastActivityAt) / 60_000;
      if (ageMin >= 15) {
        console.error(
          `[runDailyRamp] STALLED waveRun runId=${top.runId} status=${top.status} (last activity ${ageMin.toFixed(1)}min ago) — operator must resume (resumeStalledWaveRun / resumeFinalizeWaveRun) or discard (discardWaveRun) before next tick.`,
        );
        return { status: "stalled-wave-run", detail: top.runId };
      }
      console.log(
        `[runDailyRamp] wave already in flight (runId=${top.runId} status=${top.status}, last activity ${ageMin.toFixed(1)}min ago) — skip`,
      );
      return { status: "wave-in-flight", detail: top.runId };
    }

    // Belt-and-suspenders lease check. `_listInFlightWaveRuns` only returns
    // runs in active states (picking/segment-created/pushing/broadcast-created)
    // — it does NOT include `failed` runs. But `failed` runs may STILL hold
    // the lease (e.g. `persist-failed`, `segment-create-failed`,
    // `batch-failure-rate-exceeded` all preserve the lease until the

View on GitHub (pinned to ffec79ac33)

Solutions

  1. Treat as a code defect: review the surrounding inFlight length check and index logic for a regression.
  2. If seen in production, capture the waveRuns query result and report — it indicates a logic bug, not an operational condition.
  3. No operator recovery action is appropriate; fix the code.
Defensive patterns

Strategy: try-catch

Try / catch

// Invariant guard — should never fire; if it does, capture diagnostics
try {
  // ... the in-flight read path ...
} catch (e) {
  if ((e as Error).message.includes('inFlight bounds-check invariant violated')) {
    // capture full waveRuns state and file a bug — this is a code defect, not an op condition
    console.error('[runDailyRamp] INVARIANT BROKEN — dump waveRuns and report a bug');
  }
  throw e;
}

Prevention

When it happens

Trigger: A future code change breaks the inFlight.length > 0 guard without updating this read; an exotic concurrency/retry artifact where the array mutated between the check and the index; effectively unreachable in correct code.

Common situations: Code regression introduced during a refactor of the in-flight guard; should never occur in a correct build.

Related errors

AI-assisted analysis of koala73/worldmonitor@ffec79ac33 (2026-08-12). Data as JSON: /api/errors/2e61d209ec2e6218. Report an issue: GitHub.