What causes "Redis snapshot lock failed" in koala73/worldmonitor?

Retry the health request — the lock acquisition is retried inside the wait loop only when the result is non-error and not 'OK'; a hard error short-circuits, so the next request re-attempts.

koala73/worldmonitor · critical

Redis snapshot lock failed

Error message

Redis snapshot lock failed

What it means

Thrown when the SET NX pipeline that acquires the health-verdict refresh lock returns null or its first entry carries an `error` field. The lock (key HEALTH_VERDICT_REFRESH_LOCK_KEY, 30s TTL via HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS) serializes cold-burst refresh so only one edge invocation runs the ~390-command sweep. Like the other snapshot-path Redis failures, this falls into the catch and returns 503 REDIS_DOWN.

Source

Thrown at api/health.js:2650

    const cachedSnapshot = parseHealthVerdictSnapshot(snapshotResult[0]?.result, snapshotNow(), { requireChecks: !compact });
    // Activation deadlines are exact to the second, so the 60s verdict cache
    // must not outlive either rollout grace. A snapshot written just before a
    // deadline would otherwise keep serving a softened verdict for up to a
    // minute after strictness was supposed to begin. Sweep fresh instead.
    if (cachedSnapshot && !hasExpiredActivationGrace(cachedSnapshot, snapshotNow())) {
      return healthResponse(cachedSnapshot, compact, headers);
    }

    refreshLockToken = `${now}:${crypto.randomUUID()}`;
    let lockResult = await redisPipeline([[
      'SET',
      HEALTH_VERDICT_REFRESH_LOCK_KEY,
      refreshLockToken,
      'EX',
      String(HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS),
      'NX',
    ]], 4_000);
    if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock failed');
    ownsSnapshotRefreshLock = lockResult[0]?.result === 'OK';

    if (!ownsSnapshotRefreshLock) {
      // Another edge invocation is already refreshing. Wait briefly for its
      // snapshot instead of multiplying the ~390-command sweep during a cold
      // burst. If the holder dies, retry SET NX until its lease expires; only
      // a lock owner may proceed to the sweep.
      const waitDeadline = Date.now() + HEALTH_VERDICT_REFRESH_WAIT_MS;
      for (
        let attempt = 0;
        attempt < HEALTH_VERDICT_REFRESH_WAIT_ATTEMPTS && Date.now() < waitDeadline;
        attempt++
      ) {
        const backoffMs = Math.min(100 * (2 ** attempt), 1_000);
        const jitterMs = Math.floor(Math.random() * 50);
        const sleepMs = Math.min(backoffMs + jitterMs, Math.max(0, waitDeadline - Date.now()));
        await new Promise((resolve) => setTimeout(resolve, sleepMs));
        const remainingMs = waitDeadline - Date.now();

View on GitHub (pinned to ffec79ac33)

Solutions

Retry the health request — the lock acquisition is retried inside the wait loop only when the result is non-error and not 'OK'; a hard error short-circuits, so the next request re-attempts.
Check Upstash for OOM or WRONGTYPE events on the HEALTH_VERDICT_REFRESH_LOCK_KEY.
If the key type is wrong, flush the lock key (`DEL health:verdict:refresh-lock` or equivalent) so SET NX can recreate it as a string.
Confirm no other code path writes a non-string to the lock key.

Example fix

// before
if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock failed');
// after — degrade gracefully: if the lock acquire errors but Redis is up,
// fall through to one direct sweep instead of hard-downing
if (!lockResult || lockResult[0]?.error) {
  // proceed without the lock; see the 'fall back to one direct sweep' comment below
}

Defensive patterns

Strategy: try-catch

Validate before calling

import { redisPipeline, getRedisCredentials } from './_upstash-json.js';

async function canAcquireLock(key: string, ttl: number): Promise<boolean> {
  if (!getRedisCredentials()) return false;
  const r = await redisPipeline([['SET', key, 'probe', 'EX', String(ttl), 'NX']], 4_000);
  return r !== null && !r[0]?.error;
}

Try / catch

// health.js catches this and returns 503 REDIS_DOWN. A more lenient variant
// matches the existing 'fall back to one direct sweep' comment:
try {
  lockResult = await redisPipeline([['SET', key, token, 'EX', ttl, 'NX']], 4_000);
} catch {
  // Redis blip on lock-acquire: do not hard-down; proceed to one bounded sweep.
  lockResult = null;
}
if (!lockResult || lockResult[0]?.error) {
  // proceed without ownership (rare path)
}

Prevention

Keep the lock key a string type — never let another writer overwrite it with a different type (WRONGTYPE surfaces as error here).
Set the lock TTL shorter than your monitor's alert threshold so a dead holder self-clears (current: 30s).
Treat lock-acquire errors as Redis-outage signals; check Upstash before assuming a logic bug.

When it happens

Trigger: After a snapshot miss, the SET NX lock-acquire pipeline fails (returns null) or Upstash reports a per-command error on the SET — e.g. WRONGTYPE if the key was overwritten by a different writer, OOM, or a transient Upstash 5xx mid-pipeline.

Common situations: Upstash briefly degraded right at the moment a cold refresh was needed; a key-type collision from a buggy writer; Upstash cluster failover causing a one-off pipeline error.

Related errors

AI-assisted analysis of koala73/worldmonitor@ffec79ac33 (2026-08-12). Data as JSON: /api/errors/000fc3e576b5da92. Report an issue: GitHub.