{"record":{"id":"93e8e959ba65641e","repo":"koala73/worldmonitor","slug":"recordwavesent-lease-lost-expected-runid-arg","errorCode":null,"errorMessage":"[_recordWaveSent] lease lost: expected runId=${args.runId}, found ${row.pendingRunId ?? \"<cleared>\"}. Refusing to advance tier — investigate what cleared the lease.","messagePattern":"\\[_recordWaveSent\\] lease lost: expected runId=(.+?), found (.+?)\\. Refusing to advance tier — investigate what cleared the lease\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"convex/broadcast/rampRunner.ts","lineNumber":792,"sourceCode":"    waveLabel: v.string(),\n    broadcastId: v.string(),\n    segmentId: v.string(),\n    assigned: v.number(),\n    sentAt: v.number(),\n  },\n  handler: async (ctx, args) => {\n    const row = await loadConfig(ctx);\n    if (!row) throw new Error(\"[_recordWaveSent] no ramp configured\");\n    if (row.currentTier !== args.expectedCurrentTier) {\n      throw new Error(\n        `[_recordWaveSent] tier moved underneath us: expected ${args.expectedCurrentTier}, found ${row.currentTier}. Refusing to overwrite.`,\n      );\n    }\n    if (row.pendingRunId !== args.runId) {\n      // The lease changed under us — operator force-released it, or\n      // recoverFromPartialFailure cleared it. We must NOT advance the tier;\n      // bubble to Convex auto-Sentry so ops can investigate.\n      throw new Error(\n        `[_recordWaveSent] lease lost: expected runId=${args.runId}, found ${row.pendingRunId ?? \"<cleared>\"}. Refusing to advance tier — investigate what cleared the lease.`,\n      );\n    }\n    await ctx.db.patch(row._id, {\n      currentTier: args.newTier,\n      lastWaveLabel: args.waveLabel,\n      lastWaveBroadcastId: args.broadcastId,\n      lastWaveSegmentId: args.segmentId,\n      lastWaveAssigned: args.assigned,\n      lastWaveSentAt: args.sentAt,\n      lastRunStatus: \"succeeded\",\n      lastRunAt: Date.now(),\n      lastRunError: undefined,\n      pendingRunId: undefined,\n      pendingRunStartedAt: undefined,\n      // Clear all per-step progress markers — this run's state is now in\n      // the lastWave* fields and the markers would otherwise leak into the\n      // next run's recovery surface.","sourceCodeStart":774,"sourceCodeEnd":810,"githubUrl":"https://github.com/koala73/worldmonitor/blob/ffec79ac339946fd2d24e85845da5755dcaa534b/convex/broadcast/rampRunner.ts#L774-L810","documentation":"_recordWaveSent additionally asserts pendingRunId === runId before advancing the tier. Even if the tier is unchanged, a lease mismatch means ownership moved (forceReleaseLease or recoverFromPartialFailure cleared it). Advancing the tier would race with whatever the new owner is doing, so it refuses and bubbles to Sentry.","triggerScenarios":"Operator forceReleaseLease'd mid-flight; recoverFromPartialFailure cleared pendingRunId; a second claim overwrote the lease while the tier (by coincidence) had not yet advanced.","commonSituations":"Operator intervened on a slow run; a recovery path cleared the lease; concurrent run reclaimed the lease.","solutions":["Investigate what cleared the lease and whether the wave actually sent before any action.","Reconcile via recoverFromPartialFailure (manual-finished if sent) rather than re-committing.","Avoid operator interventions overlapping live runs."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Runner-side: re-verify lease before terminal commit\nconst row = await ctx.db.query('broadcastRampConfig').withIndex('by_key', (q) => q.eq('key', 'current')).first();\nif (!row || row.pendingRunId !== runId) {\n  return { aborted: 'lease-lost' };\n}","typeGuard":"function leaseIsOurs(row: { pendingRunId?: string } | null, runId: string): row is { pendingRunId: string } {\n  return row !== null && row.pendingRunId === runId;\n}","tryCatchPattern":"try {\n  await ctx.runMutation(internal.broadcast.rampRunner._recordWaveSent, { runId, ... });\n} catch (e) {\n  if ((e as Error).message.includes('lease lost')) {\n    return { aborted: 'lease-lost', detail: (e as Error).message };\n  }\n  throw e;\n}","preventionTips":["Reserve lease clearing for forceReleaseLease; do not let recovery paths overlap live runs.","Treat lease-lost at commit as terminal — reconcile, do not retry.","Confirm whether the broadcast actually sent before deciding recovery branch."],"tags":["convex","broadcast-ramp","lease","concurrency","runner-internal"],"backgroundTag":null,"analyzedSha":"ffec79ac339946fd2d24e85845da5755dcaa534b","analyzedAt":"2026-08-12T11:24:56.012Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}