{"record":{"id":"d476212cec199fd4","repo":"koala73/worldmonitor","slug":"failed-to-persist-resilience-score-generation-for","errorCode":null,"errorMessage":"Failed to persist resilience score generation for ${generation.trace.countryCode}","messagePattern":"Failed to persist resilience score generation for (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"server/worldmonitor/resilience/v1/_shared.ts","lineNumber":1105,"sourceCode":"    scoreTraceCacheKey(generation.trace.countryCode, generation.trace.generationId),\n    generation.trace,\n    RESILIENCE_SCORE_TRACE_CACHE_TTL_SECONDS,\n  );\n  if (!persisted) {\n    throw new Error(`Failed to persist resilience trace generation for ${generation.trace.countryCode}`);\n  }\n}\n\nasync function persistFullScoreGeneration(generation: ResilienceScoreGeneration): Promise<CachedScorePayload> {\n  await persistGenerationTrace(generation);\n  const cachedPayload = cachedPayloadForGeneration(generation);\n  const scorePersisted = await setCachedJson(\n    scoreCacheKey(generation.trace.countryCode),\n    cachedPayload,\n    RESILIENCE_SCORE_CACHE_TTL_SECONDS,\n  );\n  if (!scorePersisted) {\n    throw new Error(`Failed to persist resilience score generation for ${generation.trace.countryCode}`);\n  }\n  return cachedPayload;\n}\n\nasync function toPublicCachedScore(\n  countryCode: string,\n  cached: CachedScorePayload,\n): Promise<GetResilienceScoreResponse> {\n  let payload = stripCacheMeta(cached);\n  const scoreInterval = await readScoreInterval(countryCode);\n  if (scoreInterval) payload = { ...payload, scoreInterval };\n\n  // The cache stores the v2 superset. Gate the public shape at serve time so a\n  // schema rollback takes effect without waiting for the score TTL.\n  if (!RESILIENCE_SCHEMA_V2_ENABLED) {\n    payload.pillars = [];\n    payload.schemaVersion = '1.0';\n  }","sourceCodeStart":1087,"sourceCodeEnd":1123,"githubUrl":"https://github.com/koala73/worldmonitor/blob/9361220cc013571781071f0206e4d80fd14b2f7f/server/worldmonitor/resilience/v1/_shared.ts#L1087-L1123","documentation":"This error is thrown in server/worldmonitor/resilience/v1/_shared.ts when the resilience score generation payload could not be written to the shared cache via setCachedJson(). The library treats a cache write failure as fatal for score generation because subsequent reads rely on the cached payload being present under scoreCacheKey(countryCode); returning a score without persisting it would break trace-consistency guarantees. It wraps the underlying cache backend failure (Redis/Upstash outage, TTL/storage error) into an explicit generation error.","triggerScenarios":"Calling ensureResilienceScoreCached (or any path that builds a fresh score generation) when setCachedJson(scoreCacheKey(...), cachedPayload, RESILIENCE_SCORE_CACHE_TTL_SECONDS) returns false — i.e., the cache client returned a null/error result on SET, typically during a Redis/Upstash outage or when the cached payload exceeds backend size limits.","commonSituations":"1) Cache service (Upstash Redis) temporarily down or rate-limited while the score cache is cold; 2) serialized cachedPayload exceeds the cache backend's max value size after schema-v2 payloads grew; 3) misconfigured or expired cache credentials in the server environment; 4) transient network partition between the Railway server and the cache.","solutions":["Check the cache backend (Redis/Upstash) is reachable and credentials are valid; retry once the cache recovers since this is a transient infrastructure failure.","Verify RESILIENCE_SCORE_CACHE_TTL_SECONDS and cache env configuration (keys/endpoints) are set correctly for the deployment.","Inspect cachedPayload size; if it exceeds the backend limit, trim the trace sidecar or raise the backend's value-size limit.","Wrap the score generation call in a retry with backoff, and surface a degraded response instead of failing the request while the cache is unavailable."],"exampleFix":"// before\nconst scorePersisted = await setCachedJson(scoreCacheKey(generation.trace.countryCode), cachedPayload, RESILIENCE_SCORE_CACHE_TTL_SECONDS);\nif (!scorePersisted) {\n  throw new Error(`Failed to persist resilience score generation for ${generation.trace.countryCode}`);\n}\n// after\nconst scorePersisted = await setCachedJson(scoreCacheKey(generation.trace.countryCode), cachedPayload, RESILIENCE_SCORE_CACHE_TTL_SECONDS);\nif (!scorePersisted) {\n  console.warn(`score cache write failed for ${generation.trace.countryCode}; serving uncacheable response`);\n  return cachedPayload; // degrade gracefully instead of throwing\n}","handlingStrategy":"try-catch","validationCode":"// preflight cache reachability\nconst cacheOk = await checkCacheHealth(); // e.g. PING the Redis client\nif (!cacheOk) console.warn('cache unavailable; score generation may fail');","typeGuard":null,"tryCatchPattern":"try {\n  const score = await ensureResilienceScoreCached(cc);\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith('Failed to persist resilience score generation')) {\n    // transient cache outage: retry with backoff or serve degraded response\n    await sleep(RETRY_MS);\n    return serveDegraded(cc);\n  }\n  throw err;\n}","preventionTips":["Monitor cache backend health/latency and alert before it saturates","Keep RESILIENCE_SCORE_CACHE_TTL_SECONDS and cache env config in one validated module","Cap payload size (trim trace sidecars) under the backend value limit","Retry transient SET failures with backoff inside the cache helper"],"tags":["cache","redis","persistence","server"],"backgroundTag":"cache-write-failed","analyzedSha":"9361220cc013571781071f0206e4d80fd14b2f7f","analyzedAt":"2026-09-01T10:32:37.851Z","contentChangedAt":"2026-09-01T10:32:37.851Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}