{"record":{"id":"a6d7d38b58868d65","repo":"koala73/worldmonitor","slug":"redis-snapshot-lock-retry-failed","errorCode":null,"errorMessage":"Redis snapshot lock retry failed","messagePattern":"Redis snapshot lock retry failed","errorType":"exception","errorClass":null,"httpStatus":503,"severity":"critical","filePath":"api/health.js","lineNumber":2689,"sourceCode":"        const refreshedResult = await redisPipeline([['GET', snapshotKey]], redisTimeoutMs);\n        if (!refreshedResult || refreshedResult[0]?.error) throw new Error('Redis snapshot wait failed');\n        const refreshedSnapshot = parseHealthVerdictSnapshot(refreshedResult[0]?.result, snapshotNow(), { requireChecks: !compact });\n        if (\n          refreshedSnapshot\n          && !hasExpiredActivationGrace(refreshedSnapshot, snapshotNow())\n        ) {\n          return healthResponse(refreshedSnapshot, compact, headers);\n        }\n\n        lockResult = await redisPipeline([[\n          'SET',\n          HEALTH_VERDICT_REFRESH_LOCK_KEY,\n          refreshLockToken,\n          'EX',\n          String(HEALTH_VERDICT_REFRESH_LOCK_TTL_SECONDS),\n          'NX',\n        ]], redisTimeoutMs);\n        if (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock retry failed');\n        ownsSnapshotRefreshLock = lockResult[0]?.result === 'OK';\n        if (ownsSnapshotRefreshLock) break;\n      }\n      // Redis stayed reachable but another refresher held the lock through our\n      // request budget. Fall back to one direct sweep rather than mislabeling\n      // healthy Redis as REDIS_DOWN. This path is bounded and should be rare;\n      // the normal cold-burst path still permits only the elected owner.\n    }\n  } catch (err) {\n    if (ownsSnapshotRefreshLock) await releaseHealthVerdictRefreshLock(refreshLockToken);\n    return jsonResponse({\n      status: 'REDIS_DOWN',\n      error: err.message,\n      checkedAt: new Date(now).toISOString(),\n    }, 503, headers);\n  }\n\n  const allDataKeys = [","sourceCodeStart":2671,"sourceCodeEnd":2707,"githubUrl":"https://github.com/koala73/worldmonitor/blob/ffec79ac339946fd2d24e85845da5755dcaa534b/api/health.js#L2671-L2707","documentation":"Thrown inside the same wait loop when a SET NX retry (attempting to steal the lock if the prior holder died) returns null or a per-command error. The retry uses the shrinking redisTimeoutMs budget; a null/error here is treated as a Redis outage and bubbles to the 503 REDIS_DOWN catch.","triggerScenarios":"During the 3s wait window, the loop re-attempts SET NX to take over the lock if the holder's lease lapsed; the SET NX pipeline returns null (timeout/HTTP error) or Upstash returns a per-command error on the SET.","commonSituations":"Upstash transient error mid-wait; the shrinking per-attempt timeout clipped the SET NX; an Upstash failover; token revoked mid-request.","solutions":["Retry the health request after a few seconds.","Verify the Upstash REST token is still valid and the URL is correct.","Check Upstash dashboards for errors around the lock key (HEALTH_VERDICT_REFRESH_LOCK_KEY).","If the lock key is in a wrong type state, DEL it so SET NX can succeed on retry."],"exampleFix":"// before\nif (!lockResult || lockResult[0]?.error) throw new Error('Redis snapshot lock retry failed');\n// after — break the loop and fall through to the bounded direct sweep\nif (!lockResult || lockResult[0]?.error) break;","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Same pattern as [11]: break the loop and fall through to the bounded direct\n// sweep instead of throwing.\nlockResult = await redisPipeline([['SET', key, token, 'EX', ttl, 'NX']], redisTimeoutMs);\nif (!lockResult || lockResult[0]?.error) break;","preventionTips":["Treat lock-retry errors as transient Redis issues, not lock-contention bugs.","Keep the per-attempt timeout reasonable so a slow SET NX does not null out.","Verify no other writer changes the type of HEALTH_VERDICT_REFRESH_LOCK_KEY."],"tags":["redis","health","distributed-lock","upstash","retryable"],"backgroundTag":null,"analyzedSha":"ffec79ac339946fd2d24e85845da5755dcaa534b","analyzedAt":"2026-08-12T11:24:56.012Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}