{"record":{"id":"000fc3e576b5da92","repo":"koala73/worldmonitor","slug":"redis-snapshot-lock-failed","errorCode":null,"errorMessage":"Redis snapshot lock failed","messagePattern":"Redis snapshot lock failed","errorType":"exception","errorClass":null,"httpStatus":503,"severity":"critical","filePath":"api/health.js","lineNumber":2650,"sourceCode":"    const cachedSnapshot = parseHealthVerdictSnapshot(snapshotResult[0]?.result, snapshotNow(), { requireChecks: !compact });\n    // Activation deadlines are exact to the second, so the 60s verdict cache\n    // must not outlive either rollout grace. A snapshot written just before a\n    // deadline would otherwise keep serving a softened verdict for up to a\n    // minute after strictness was supposed to begin. Sweep fresh instead.\n    if (cachedSnapshot && !hasExpiredActivationGrace(cachedSnapshot, snapshotNow())) {\n      return healthResponse(cachedSnapshot, compact, headers);\n    }\n\n    refreshLockToken = `${now}:${crypto.randomUUID()}`;\n    let lockResult = await redisPipeline([[\n      'SET',\n      HEALTH_VERDICT_REFRESH_LOCK_KEY,\n      refreshLockToken,\n      'EX',\n      String(HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS),\n      'NX',\n    ]], 4_000);\n    if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock failed');\n    ownsSnapshotRefreshLock = lockResult[0]?.result === 'OK';\n\n    if (!ownsSnapshotRefreshLock) {\n      // Another edge invocation is already refreshing. Wait briefly for its\n      // snapshot instead of multiplying the ~390-command sweep during a cold\n      // burst. If the holder dies, retry SET NX until its lease expires; only\n      // a lock owner may proceed to the sweep.\n      const waitDeadline = Date.now() + HEALTH_VERDICT_REFRESH_WAIT_MS;\n      for (\n        let attempt = 0;\n        attempt < HEALTH_VERDICT_REFRESH_WAIT_ATTEMPTS && Date.now() < waitDeadline;\n        attempt++\n      ) {\n        const backoffMs = Math.min(100 * (2 ** attempt), 1_000);\n        const jitterMs = Math.floor(Math.random() * 50);\n        const sleepMs = Math.min(backoffMs + jitterMs, Math.max(0, waitDeadline - Date.now()));\n        await new Promise((resolve) => setTimeout(resolve, sleepMs));\n        const remainingMs = waitDeadline - Date.now();","sourceCodeStart":2632,"sourceCodeEnd":2668,"githubUrl":"https://github.com/koala73/worldmonitor/blob/ffec79ac339946fd2d24e85845da5755dcaa534b/api/health.js#L2632-L2668","documentation":"Thrown when the SET NX pipeline that acquires the health-verdict refresh lock returns null or its first entry carries an `error` field. The lock (key HEALTH_VERDICT_REFRESH_LOCK_KEY, 30s TTL via HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS) serializes cold-burst refresh so only one edge invocation runs the ~390-command sweep. Like the other snapshot-path Redis failures, this falls into the catch and returns 503 REDIS_DOWN.","triggerScenarios":"After a snapshot miss, the SET NX lock-acquire pipeline fails (returns null) or Upstash reports a per-command error on the SET — e.g. WRONGTYPE if the key was overwritten by a different writer, OOM, or a transient Upstash 5xx mid-pipeline.","commonSituations":"Upstash briefly degraded right at the moment a cold refresh was needed; a key-type collision from a buggy writer; Upstash cluster failover causing a one-off pipeline error.","solutions":["Retry the health request — the lock acquisition is retried inside the wait loop only when the result is non-error and not 'OK'; a hard error short-circuits, so the next request re-attempts.","Check Upstash for OOM or WRONGTYPE events on the HEALTH_VERDICT_REFRESH_LOCK_KEY.","If the key type is wrong, flush the lock key (`DEL health:verdict:refresh-lock` or equivalent) so SET NX can recreate it as a string.","Confirm no other code path writes a non-string to the lock key."],"exampleFix":"// before\nif (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock failed');\n// after — degrade gracefully: if the lock acquire errors but Redis is up,\n// fall through to one direct sweep instead of hard-downing\nif (!lockResult || lockResult[0]?.error) {\n  // proceed without the lock; see the 'fall back to one direct sweep' comment below\n}","handlingStrategy":"try-catch","validationCode":"import { redisPipeline, getRedisCredentials } from './_upstash-json.js';\n\nasync function canAcquireLock(key: string, ttl: number): Promise<boolean> {\n  if (!getRedisCredentials()) return false;\n  const r = await redisPipeline([['SET', key, 'probe', 'EX', String(ttl), 'NX']], 4_000);\n  return r !== null && !r[0]?.error;\n}","typeGuard":null,"tryCatchPattern":"// health.js catches this and returns 503 REDIS_DOWN. A more lenient variant\n// matches the existing 'fall back to one direct sweep' comment:\ntry {\n  lockResult = await redisPipeline([['SET', key, token, 'EX', ttl, 'NX']], 4_000);\n} catch {\n  // Redis blip on lock-acquire: do not hard-down; proceed to one bounded sweep.\n  lockResult = null;\n}\nif (!lockResult || lockResult[0]?.error) {\n  // proceed without ownership (rare path)\n}","preventionTips":["Keep the lock key a string type — never let another writer overwrite it with a different type (WRONGTYPE surfaces as error here).","Set the lock TTL shorter than your monitor's alert threshold so a dead holder self-clears (current: 30s).","Treat lock-acquire errors as Redis-outage signals; check Upstash before assuming a logic bug."],"tags":["redis","health","distributed-lock","upstash","cold-start"],"backgroundTag":null,"analyzedSha":"ffec79ac339946fd2d24e85845da5755dcaa534b","analyzedAt":"2026-08-12T11:24:56.012Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}