{"record":{"id":"4f6091b8a1f33e3c","repo":"koala73/worldmonitor","slug":"failed-to-persist-resilience-trace-generation-for","errorCode":null,"errorMessage":"Failed to persist resilience trace generation for ${generation.trace.countryCode}","messagePattern":"Failed to persist resilience trace generation for (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"server/worldmonitor/resilience/v1/_shared.ts","lineNumber":1092,"sourceCode":"  payload: CachedScorePayload,\n): value is ResilienceScoreTraceSidecar {\n  if (!value || typeof value !== 'object') return false;\n  const trace = value as Partial<ResilienceScoreTraceSidecar>;\n  return trace.version === 1\n    && trace.countryCode === countryCode\n    && trace.generationId === payload._traceGenerationId\n    && trace.cacheIdentity === payload._traceCacheIdentity\n    && trace.snapshot != null;\n}\n\nasync function persistGenerationTrace(generation: ResilienceScoreGeneration): Promise<void> {\n  const persisted = await setCachedJson(\n    scoreTraceCacheKey(generation.trace.countryCode, generation.trace.generationId),\n    generation.trace,\n    RESILIENCE_SCORE_TRACE_CACHE_TTL_SECONDS,\n  );\n  if (!persisted) {\n    throw new Error(`Failed to persist resilience trace generation for ${generation.trace.countryCode}`);\n  }\n}\n\nasync function persistFullScoreGeneration(generation: ResilienceScoreGeneration): Promise<CachedScorePayload> {\n  await persistGenerationTrace(generation);\n  const cachedPayload = cachedPayloadForGeneration(generation);\n  const scorePersisted = await setCachedJson(\n    scoreCacheKey(generation.trace.countryCode),\n    cachedPayload,\n    RESILIENCE_SCORE_CACHE_TTL_SECONDS,\n  );\n  if (!scorePersisted) {\n    throw new Error(`Failed to persist resilience score generation for ${generation.trace.countryCode}`);\n  }\n  return cachedPayload;\n}\n\nasync function toPublicCachedScore(","sourceCodeStart":1074,"sourceCodeEnd":1110,"githubUrl":"https://github.com/koala73/worldmonitor/blob/9361220cc013571781071f0206e4d80fd14b2f7f/server/worldmonitor/resilience/v1/_shared.ts#L1074-L1110","documentation":"persistGenerationTrace writes a resilience score trace to the cache via setCachedJson (key scoreTraceCacheKey(countryCode, generationId), TTL RESILIENCE_SCORE_TRACE_CACHE_TTL_SECONDS) and throws Error 'Failed to persist resilience trace generation for <countryCode>' when setCachedJson does not confirm success. The trace is required for later score retrieval, so a silent write failure would leave an inconsistent generation, hence fail-fast.","triggerScenarios":"setCachedJson returns a falsy result for the trace key — typically because the Redis/Upstash write failed (HTTP error, quota, timeout) or the cache helper signals failure by returning null/false instead of throwing.","commonSituations":"Upstash outage or rate limiting during score generation; expired/invalid Redis credentials in the deployment; key too large for the plan or write quota exhausted; cache helper contract change making it return null on transient errors.","solutions":["Check why setCachedJson returned falsy: inspect its Redis write path for HTTP errors, quota limits, or timeouts at that moment","Verify Redis credentials and health in the deployment environment","Make the write retryable (bounded retries with backoff) before failing the whole generation","If traces are best-effort, catch this error and degrade gracefully instead of failing score generation"],"exampleFix":"// before\nconst persisted = await setCachedJson(key, trace, TTL);\nif (!persisted) throw new Error(`Failed to persist resilience trace generation for ${cc}`);\n// after\nlet persisted = false;\nfor (let i = 0; i < 3 && !persisted; i++) {\n  persisted = await setCachedJson(key, trace, TTL);\n  if (!persisted) await sleep(100 * 2 ** i);\n}\nif (!persisted) throw new Error(`Failed to persist resilience trace generation for ${cc}`);","handlingStrategy":"retry","validationCode":"// Verify cache backend is writable before generating scores\nconst canWrite = await setCachedJson('health:probe', 'ok', 10);\nif (!canWrite) alertCacheBackendDown();","typeGuard":"function isTracePersistFailure(e: unknown): boolean {\n  return e instanceof Error && e.message.startsWith('Failed to persist resilience trace generation for ');\n}","tryCatchPattern":"try {\n  await persistGenerationTrace(generation);\n} catch (e) {\n  if (isTracePersistFailure(e)) {\n    await queueTraceForRetry(generation); // retry later or serve score without trace\n  } else throw e;\n}","preventionTips":["Check setCachedJson's return contract and log the underlying Redis error, not just falsy","Add bounded retries with backoff inside the persist path","Monitor Redis write health and quotas in production","Decide explicitly whether traces are required or best-effort, and handle accordingly"],"tags":["redis","persistence","cache-write","resilience"],"backgroundTag":"cache-write-failure","analyzedSha":"9361220cc013571781071f0206e4d80fd14b2f7f","analyzedAt":"2026-09-01T10:32:37.851Z","contentChangedAt":"2026-09-01T10:32:37.851Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}