{"record":{"id":"d68ae3a14ed02968","repo":"koala73/worldmonitor","slug":"recordwavesent-tier-moved-underneath-us-expect","errorCode":null,"errorMessage":"[_recordWaveSent] tier moved underneath us: expected ${args.expectedCurrentTier}, found ${row.currentTier}. Refusing to overwrite.","messagePattern":"\\[_recordWaveSent\\] tier moved underneath us: expected (.+?), found (.+?)\\. Refusing to overwrite\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"convex/broadcast/rampRunner.ts","lineNumber":784,"sourceCode":" * record a successful wave-send. Validates that the lease still belongs to\n * this runId AND clears all pending-progress markers.\n */\nexport const _recordWaveSent = internalMutation({\n  args: {\n    runId: v.string(),\n    expectedCurrentTier: v.number(),\n    newTier: v.number(),\n    waveLabel: v.string(),\n    broadcastId: v.string(),\n    segmentId: v.string(),\n    assigned: v.number(),\n    sentAt: v.number(),\n  },\n  handler: async (ctx, args) => {\n    const row = await loadConfig(ctx);\n    if (!row) throw new Error(\"[_recordWaveSent] no ramp configured\");\n    if (row.currentTier !== args.expectedCurrentTier) {\n      throw new Error(\n        `[_recordWaveSent] tier moved underneath us: expected ${args.expectedCurrentTier}, found ${row.currentTier}. Refusing to overwrite.`,\n      );\n    }\n    if (row.pendingRunId !== args.runId) {\n      // The lease changed under us — operator force-released it, or\n      // recoverFromPartialFailure cleared it. We must NOT advance the tier;\n      // bubble to Convex auto-Sentry so ops can investigate.\n      throw new Error(\n        `[_recordWaveSent] lease lost: expected runId=${args.runId}, found ${row.pendingRunId ?? \"<cleared>\"}. Refusing to advance tier — investigate what cleared the lease.`,\n      );\n    }\n    await ctx.db.patch(row._id, {\n      currentTier: args.newTier,\n      lastWaveLabel: args.waveLabel,\n      lastWaveBroadcastId: args.broadcastId,\n      lastWaveSegmentId: args.segmentId,\n      lastWaveAssigned: args.assigned,\n      lastWaveSentAt: args.sentAt,","sourceCodeStart":766,"sourceCodeEnd":802,"githubUrl":"https://github.com/koala73/worldmonitor/blob/ffec79ac339946fd2d24e85845da5755dcaa534b/convex/broadcast/rampRunner.ts#L766-L802","documentation":"_recordWaveSent asserts broadcastRampConfig.currentTier still equals the expectedCurrentTier the run was started with. A mismatch means another run already advanced (or rewound) the tier underneath this one — committing would overwrite the wrong tier's state or double-advance. The handler refuses and surfaces to Sentry.","triggerScenarios":"Two runs both passed the lease claim on the same tier (race) and one already committed; an operator used recoverFromPartialFailure which advanced currentTier; abortRamp+initRamp reset the tier while a run was in flight.","commonSituations":"Concurrent cron + manual run; a recovered partial-failure advanced the tier while the original run was still alive; the ramp was re-seeded mid-flight.","solutions":["Investigate current currentTier vs the run's expected tier — do not blindly re-commit.","If another run legitimately advanced, discard this run's result (it's stale).","If the broadcast actually sent but the tier moved, use recoverFromPartialFailure(manual-finished) to reconcile rather than re-running _recordWaveSent."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Runner-side: re-verify tier unchanged before terminal commit\nconst row = await ctx.db.query('broadcastRampConfig').withIndex('by_key', (q) => q.eq('key', 'current')).first();\nif (!row || row.currentTier !== expectedCurrentTier) {\n  return { aborted: 'tier-moved', current: row?.currentTier };\n}","typeGuard":"function tierMatches(row: { currentTier: number } | null, expected: number): row is { currentTier: number } {\n  return row !== null && row.currentTier === expected;\n}","tryCatchPattern":"try {\n  await ctx.runMutation(internal.broadcast.rampRunner._recordWaveSent, { runId, expectedCurrentTier, ... });\n} catch (e) {\n  if ((e as Error).message.includes('tier moved')) {\n    // another run advanced the tier — do not re-commit; reconcile via recoverFromPartialFailure\n    return { aborted: 'tier-moved', detail: (e as Error).message };\n  }\n  throw e;\n}","preventionTips":["Eliminate concurrent runs (cron + manual) so only one path can advance the tier.","Treat tier-mismatch as non-retriable; investigate the current tier before any action.","Avoid recoverFromPartialFailure overlapping a still-live run."],"tags":["convex","broadcast-ramp","lease","concurrency","runner-internal","race-condition"],"backgroundTag":null,"analyzedSha":"ffec79ac339946fd2d24e85845da5755dcaa534b","analyzedAt":"2026-08-12T11:24:56.012Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}