koala73/worldmonitor · error · Error

[_recordWaveSent] lease lost: expected runId=${args.runId},

Error message

[_recordWaveSent] lease lost: expected runId=${args.runId}, found ${row.pendingRunId ?? "<cleared>"}. Refusing to advance tier — investigate what cleared the lease.

What it means

_recordWaveSent additionally asserts pendingRunId === runId before advancing the tier. Even if the tier is unchanged, a lease mismatch means ownership moved (forceReleaseLease or recoverFromPartialFailure cleared it). Advancing the tier would race with whatever the new owner is doing, so it refuses and bubbles to Sentry.

Source

Thrown at convex/broadcast/rampRunner.ts:792

    waveLabel: v.string(),
    broadcastId: v.string(),
    segmentId: v.string(),
    assigned: v.number(),
    sentAt: v.number(),
  },
  handler: async (ctx, args) => {
    const row = await loadConfig(ctx);
    if (!row) throw new Error("[_recordWaveSent] no ramp configured");
    if (row.currentTier !== args.expectedCurrentTier) {
      throw new Error(
        `[_recordWaveSent] tier moved underneath us: expected ${args.expectedCurrentTier}, found ${row.currentTier}. Refusing to overwrite.`,
      );
    }
    if (row.pendingRunId !== args.runId) {
      // The lease changed under us — operator force-released it, or
      // recoverFromPartialFailure cleared it. We must NOT advance the tier;
      // bubble to Convex auto-Sentry so ops can investigate.
      throw new Error(
        `[_recordWaveSent] lease lost: expected runId=${args.runId}, found ${row.pendingRunId ?? "<cleared>"}. Refusing to advance tier — investigate what cleared the lease.`,
      );
    }
    await ctx.db.patch(row._id, {
      currentTier: args.newTier,
      lastWaveLabel: args.waveLabel,
      lastWaveBroadcastId: args.broadcastId,
      lastWaveSegmentId: args.segmentId,
      lastWaveAssigned: args.assigned,
      lastWaveSentAt: args.sentAt,
      lastRunStatus: "succeeded",
      lastRunAt: Date.now(),
      lastRunError: undefined,
      pendingRunId: undefined,
      pendingRunStartedAt: undefined,
      // Clear all per-step progress markers — this run's state is now in
      // the lastWave* fields and the markers would otherwise leak into the
      // next run's recovery surface.

View on GitHub (pinned to ffec79ac33)

Solutions

  1. Investigate what cleared the lease and whether the wave actually sent before any action.
  2. Reconcile via recoverFromPartialFailure (manual-finished if sent) rather than re-committing.
  3. Avoid operator interventions overlapping live runs.
Defensive patterns

Strategy: try-catch

Validate before calling

// Runner-side: re-verify lease before terminal commit
const row = await ctx.db.query('broadcastRampConfig').withIndex('by_key', (q) => q.eq('key', 'current')).first();
if (!row || row.pendingRunId !== runId) {
  return { aborted: 'lease-lost' };
}

Type guard

function leaseIsOurs(row: { pendingRunId?: string } | null, runId: string): row is { pendingRunId: string } {
  return row !== null && row.pendingRunId === runId;
}

Try / catch

try {
  await ctx.runMutation(internal.broadcast.rampRunner._recordWaveSent, { runId, ... });
} catch (e) {
  if ((e as Error).message.includes('lease lost')) {
    return { aborted: 'lease-lost', detail: (e as Error).message };
  }
  throw e;
}

Prevention

When it happens

Trigger: Operator forceReleaseLease'd mid-flight; recoverFromPartialFailure cleared pendingRunId; a second claim overwrote the lease while the tier (by coincidence) had not yet advanced.

Common situations: Operator intervened on a slow run; a recovery path cleared the lease; concurrent run reclaimed the lease.

Related errors

AI-assisted analysis of koala73/worldmonitor@ffec79ac33 (2026-08-12). Data as JSON: /api/errors/93e8e959ba65641e. Report an issue: GitHub.