koala73/worldmonitor · critical

Redis snapshot lock retry failed

Error message

Redis snapshot lock retry failed

What it means

Thrown inside the same wait loop when a SET NX retry (attempting to steal the lock if the prior holder died) returns null or a per-command error. The retry uses the shrinking redisTimeoutMs budget; a null/error here is treated as a Redis outage and bubbles to the 503 REDIS_DOWN catch.

Source

Thrown at api/health.js:2689

        const refreshedResult = await redisPipeline([['GET', snapshotKey]], redisTimeoutMs);
        if (!refreshedResult || refreshedResult[0]?.error) throw new Error('Redis snapshot wait failed');
        const refreshedSnapshot = parseHealthVerdictSnapshot(refreshedResult[0]?.result, snapshotNow(), { requireChecks: !compact });
        if (
          refreshedSnapshot
          && !hasExpiredActivationGrace(refreshedSnapshot, snapshotNow())
        ) {
          return healthResponse(refreshedSnapshot, compact, headers);
        }

        lockResult = await redisPipeline([[
          'SET',
          HEALTH_VERDICT_REFRESH_LOCK_KEY,
          refreshLockToken,
          'EX',
          String(HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS),
          'NX',
        ]], redisTimeoutMs);
        if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock retry failed');
        ownsSnapshotRefreshLock = lockResult[0]?.result === 'OK';
        if (ownsSnapshotRefreshLock) break;
      }
      // Redis stayed reachable but another refresher held the lock through our
      // request budget. Fall back to one direct sweep rather than mislabeling
      // healthy Redis as REDIS_DOWN. This path is bounded and should be rare;
      // the normal cold-burst path still permits only the elected owner.
    }
  } catch (err) {
    if (ownsSnapshotRefreshLock) await releaseHealthVerdictRefreshLock(refreshLockToken);
    return jsonResponse({
      status: 'REDIS_DOWN',
      error: err.message,
      checkedAt: new Date(now).toISOString(),
    }, 503, headers);
  }

  const allDataKeys = [

View on GitHub (pinned to ffec79ac33)

Solutions

1. Retry the health request after a few seconds.
2. Verify the Upstash REST token is still valid and the URL is correct.
3. Check Upstash dashboards for errors around the lock key (HEALTH_VERDICT_REFRESH_LOCK_KEY).
4. If the lock key is in a wrong type state, DEL it so SET NX can succeed on retry.

Example fix

// before
if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock retry failed');
// after — break the loop and fall through to the bounded direct sweep
if (!lockResult || lockResult[0]?.error) break;

// Same pattern as [11]: break the loop and fall through to the bounded direct
// sweep instead of throwing.
lockResult = await redisPipeline([['SET', key, token, 'EX', ttl, 'NX']], redisTimeoutMs);
if (!lockResult || lockResult[0]?.error) break;

When it happens

Trigger: During the 3s wait window, the loop re-attempts SET NX to take over the lock if the holder's lease lapsed; the SET NX pipeline returns null (timeout/HTTP error) or Upstash returns a per-command error on the SET.

Common situations: Upstash transient error mid-wait; the shrinking per-attempt timeout clipped the SET NX; an Upstash failover; token revoked mid-request.

Related errors

• Redis snapshot lock failed
• Redis snapshot wait failed
• Redis request failed
• Redis not configured
• Webhook URL DNS resolution failed: ${message}

AI-assisted analysis of koala73/worldmonitor@ffec79ac33 (2026-08-12). Data as JSON: /api/errors/a6d7d38b58868d65. Report an issue: GitHub.