koala73/worldmonitor · critical

Redis snapshot wait failed

Error message

Redis snapshot wait failed

What it means

Thrown inside the wait-for-another-refresher loop when a GET snapshot retry (during the bounded 3s wait window, up to HEALTH_VERDICT_REFRESH_WAIT_ATTEMPTS attempts with exponential backoff) returns null or a per-command error. It indicates Redis became unreachable mid-wait — distinct from the lock being held, which simply continues looping.

Source

Thrown at api/health.js:2672

      // Another edge invocation is already refreshing. Wait briefly for its
      // snapshot instead of multiplying the ~390-command sweep during a cold
      // burst. If the holder dies, retry SET NX until its lease expires; only
      // a lock owner may proceed to the sweep.
      const waitDeadline = Date.now() + HEALTH_VERDICT_REFRESH_WAIT_MS;
      for (
        let attempt = 0;
        attempt < HEALTH_VERDICT_REFRESH_WAIT_ATTEMPTS && Date.now() < waitDeadline;
        attempt++
      ) {
        const backoffMs = Math.min(100 * (2 ** attempt), 1_000);
        const jitterMs = Math.floor(Math.random() * 50);
        const sleepMs = Math.min(backoffMs + jitterMs, Math.max(0, waitDeadline - Date.now()));
        await new Promise((resolve) => setTimeout(resolve, sleepMs));
        const remainingMs = waitDeadline - Date.now();
        if (remainingMs < HEALTH_VERDICT_MIN_REDIS_TIMEOUT_MS) break;
        const redisTimeoutMs = Math.min(4_000, remainingMs);
        const refreshedResult = await redisPipeline([['GET', snapshotKey]], redisTimeoutMs);
        if (!refreshedResult || refreshedResult[0]?.error) throw new Error('Redis snapshot wait failed');
        const refreshedSnapshot = parseHealthVerdictSnapshot(refreshedResult[0]?.result, snapshotNow(), { requireChecks: !compact });
        if (
          refreshedSnapshot
          && !hasExpiredActivationGrace(refreshedSnapshot, snapshotNow())
        ) {
          return healthResponse(refreshedSnapshot, compact, headers);
        }

        lockResult = await redisPipeline([[
          'SET',
          HEALTH_VERDICT_REFRESH_LOCK_KEY,
          refreshLockToken,
          'EX',
          String(HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS),
          'NX',
        ]], redisTimeoutMs);
        if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock retry failed');
        ownsSnapshotRefreshLock = lockResult[0]?.result === 'OK';

View on GitHub (pinned to ffec79ac33)

Solutions

1. Retry the health request — the next invocation typically finds a fresh snapshot already written by the lock owner.
2. Investigate Upstash latency around the incident window; if per-attempt timeouts are firing, the snapshot read budget or DB plan may need adjusting.
3. Confirm the Upstash token and URL are still valid (a revoked token turns every pipeline into null).
4. If this fires repeatedly, the refresh lock may be stuck (holder died without releasing) — verify the lock TTL (30s) is shorter than your monitor's alert threshold.

Example fix

// before
if (!refreshedResult || refreshedResult[0]?.error) throw new Error('Redis snapshot wait failed');
// after — break out of the wait loop and fall through to the bounded direct
// sweep instead of hard-downing, matching the 'rare path' comment below
if (!refreshedResult || refreshedResult[0]?.error) break;

// health.js catches and returns 503. To match the 'rare path' fallback intent,
// break the wait loop on a Redis error rather than throwing:
const refreshedResult = await redisPipeline([['GET', snapshotKey]], redisTimeoutMs);
if (!refreshedResult || refreshedResult[0]?.error) break; // fall through to direct sweep

When it happens

Trigger: A second edge invocation arrives while another owns the refresh lock; it polls the snapshot key with backoff, and one of those GET pipelines returns null (Upstash timeout/error) — api/health.js:2672 throws, the catch returns 503 REDIS_DOWN.

Common situations: Redis degrades partway through a cold-burst window; Upstash latency spikes so the shrinking per-attempt timeout (min(4s, remainingMs)) starts timing out; transient connectivity loss between Edge and Upstash.

Related errors

• Redis snapshot lock failed
• Redis snapshot lock retry failed
• Redis request failed
• Redis not configured
• No event feeds are available for exposure enrichment

AI-assisted analysis of koala73/worldmonitor@ffec79ac33 (2026-08-12). Data as JSON: /api/errors/20270c61c8510b28. Report an issue: GitHub.