{"record":{"id":"20270c61c8510b28","repo":"koala73/worldmonitor","slug":"redis-snapshot-wait-failed","errorCode":null,"errorMessage":"Redis snapshot wait failed","messagePattern":"Redis snapshot wait failed","errorType":"exception","errorClass":null,"httpStatus":503,"severity":"critical","filePath":"api/health.js","lineNumber":2672,"sourceCode":"      // Another edge invocation is already refreshing. Wait briefly for its\n      // snapshot instead of multiplying the ~390-command sweep during a cold\n      // burst. If the holder dies, retry SET NX until its lease expires; only\n      // a lock owner may proceed to the sweep.\n      const waitDeadline = Date.now() + HEALTH_VERDICT_REFRESH_WAIT_MS;\n      for (\n        let attempt = 0;\n        attempt < HEALTH_VERDICT_REFRESH_WAIT_ATTEMPTS && Date.now() < waitDeadline;\n        attempt++\n      ) {\n        const backoffMs = Math.min(100 * (2 ** attempt), 1_000);\n        const jitterMs = Math.floor(Math.random() * 50);\n        const sleepMs = Math.min(backoffMs + jitterMs, Math.max(0, waitDeadline - Date.now()));\n        await new Promise((resolve) => setTimeout(resolve, sleepMs));\n        const remainingMs = waitDeadline - Date.now();\n        if (remainingMs < HEALTH_VERDICT_MIN_REDIS_TIMEOUT_MS) break;\n        const redisTimeoutMs = Math.min(4_000, remainingMs);\n        const refreshedResult = await redisPipeline([['GET', snapshotKey]], redisTimeoutMs);\n        if (!refreshedResult || refreshedResult[0]?.error) throw new Error('Redis snapshot wait failed');\n        const refreshedSnapshot = parseHealthVerdictSnapshot(refreshedResult[0]?.result, snapshotNow(), { requireChecks: !compact });\n        if (\n          refreshedSnapshot\n          && !hasExpiredActivationGrace(refreshedSnapshot, snapshotNow())\n        ) {\n          return healthResponse(refreshedSnapshot, compact, headers);\n        }\n\n        lockResult = await redisPipeline([[\n          'SET',\n          HEALTH_VERDICT_REFRESH_LOCK_KEY,\n          refreshLockToken,\n          'EX',\n          String(HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS),\n          'NX',\n        ]], redisTimeoutMs);\n        if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock retry failed');\n        ownsSnapshotRefreshLock = lockResult[0]?.result === 'OK';","sourceCodeStart":2654,"sourceCodeEnd":2690,"githubUrl":"https://github.com/koala73/worldmonitor/blob/ffec79ac339946fd2d24e85845da5755dcaa534b/api/health.js#L2654-L2690","documentation":"Thrown inside the wait-for-another-refresher loop when a GET snapshot retry (during the bounded 3s wait window, up to HEALTH_VERDICT_REFRESH_WAIT_ATTEMPTS attempts with exponential backoff) returns null or a per-command error. It indicates Redis became unreachable mid-wait — distinct from the lock being held, which simply continues looping.","triggerScenarios":"A second edge invocation arrives while another owns the refresh lock; it polls the snapshot key with backoff, and one of those GET pipelines returns null (Upstash timeout/error) — api/health.js:2672 throws, the catch returns 503 REDIS_DOWN.","commonSituations":"Redis degrades partway through a cold-burst window; Upstash latency spikes so the shrinking per-attempt timeout (min(4s, remainingMs)) starts timing out; transient connectivity loss between Edge and Upstash.","solutions":["Retry the health request — the next invocation typically finds a fresh snapshot already written by the lock owner.","Investigate Upstash latency around the incident window; if per-attempt timeouts are firing, the snapshot read budget or DB plan may need adjusting.","Confirm the Upstash token and URL are still valid (a revoked token turns every pipeline into null).","If this fires repeatedly, the refresh lock may be stuck (holder died without releasing) — verify the lock TTL (30s) is shorter than your monitor's alert threshold."],"exampleFix":"// before\nif (!refreshedResult || refreshedResult[0]?.error) throw new Error('Redis snapshot wait failed');\n// after — break out of the wait loop and fall through to the bounded direct\n// sweep instead of hard-downing, matching the 'rare path' comment below\nif (!refreshedResult || refreshedResult[0]?.error) break;","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// health.js catches and returns 503. To match the 'rare path' fallback intent,\n// break the wait loop on a Redis error rather than throwing:\nconst refreshedResult = await redisPipeline([['GET', snapshotKey]], redisTimeoutMs);\nif (!refreshedResult || refreshedResult[0]?.error) break; // fall through to direct sweep","preventionTips":["Monitor the wait-loop path — frequent fires indicate Redis cannot keep up with cold bursts.","Ensure the snapshot writer is healthy so waiters find a fresh snapshot quickly.","Keep the refresh-lock TTL (30s) tight so a dead holder does not strand waiters."],"tags":["redis","health","distributed-lock","backoff","upstash"],"backgroundTag":null,"analyzedSha":"ffec79ac339946fd2d24e85845da5755dcaa534b","analyzedAt":"2026-08-12T11:24:56.012Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}