koala73/worldmonitor · error · Error

Failed to persist resilience trace generation for ${generati

Error message

Failed to persist resilience trace generation for ${generation.trace.countryCode}

What it means

persistGenerationTrace writes a resilience score trace to the cache via setCachedJson (key scoreTraceCacheKey(countryCode, generationId), TTL RESILIENCE_SCORE_TRACE_CACHE_TTL_SECONDS) and throws Error 'Failed to persist resilience trace generation for <countryCode>' when setCachedJson does not confirm success. The trace is required for later score retrieval, so a silent write failure would leave an inconsistent generation, hence fail-fast.

Source

Thrown at server/worldmonitor/resilience/v1/_shared.ts:1092

  payload: CachedScorePayload,
): value is ResilienceScoreTraceSidecar {
  if (!value || typeof value !== 'object') return false;
  const trace = value as Partial<ResilienceScoreTraceSidecar>;
  return trace.version === 1
    && trace.countryCode === countryCode
    && trace.generationId === payload._traceGenerationId
    && trace.cacheIdentity === payload._traceCacheIdentity
    && trace.snapshot != null;
}

async function persistGenerationTrace(generation: ResilienceScoreGeneration): Promise<void> {
  const persisted = await setCachedJson(
    scoreTraceCacheKey(generation.trace.countryCode, generation.trace.generationId),
    generation.trace,
    RESILIENCE_SCORE_TRACE_CACHE_TTL_SECONDS,
  );
  if (!persisted) {
    throw new Error(`Failed to persist resilience trace generation for ${generation.trace.countryCode}`);
  }
}

async function persistFullScoreGeneration(generation: ResilienceScoreGeneration): Promise<CachedScorePayload> {
  await persistGenerationTrace(generation);
  const cachedPayload = cachedPayloadForGeneration(generation);
  const scorePersisted = await setCachedJson(
    scoreCacheKey(generation.trace.countryCode),
    cachedPayload,
    RESILIENCE_SCORE_CACHE_TTL_SECONDS,
  );
  if (!scorePersisted) {
    throw new Error(`Failed to persist resilience score generation for ${generation.trace.countryCode}`);
  }
  return cachedPayload;
}

async function toPublicCachedScore(

View on GitHub (pinned to 9361220cc0)

Solutions

Check why setCachedJson returned falsy: inspect its Redis write path for HTTP errors, quota limits, or timeouts at that moment
Verify Redis credentials and health in the deployment environment
Make the write retryable (bounded retries with backoff) before failing the whole generation
If traces are best-effort, catch this error and degrade gracefully instead of failing score generation

Example fix

// before
const persisted = await setCachedJson(key, trace, TTL);
if (!persisted) throw new Error(`Failed to persist resilience trace generation for ${cc}`);
// after
let persisted = false;
for (let i = 0; i < 3 && !persisted; i++) {
  persisted = await setCachedJson(key, trace, TTL);
  if (!persisted) await sleep(100 * 2 ** i);
}
if (!persisted) throw new Error(`Failed to persist resilience trace generation for ${cc}`);

Defensive patterns

Strategy: retry

Validate before calling

// Verify cache backend is writable before generating scores
const canWrite = await setCachedJson('health:probe', 'ok', 10);
if (!canWrite) alertCacheBackendDown();

Type guard

function isTracePersistFailure(e: unknown): boolean {
  return e instanceof Error && e.message.startsWith('Failed to persist resilience trace generation for ');
}

Try / catch

try {
  await persistGenerationTrace(generation);
} catch (e) {
  if (isTracePersistFailure(e)) {
    await queueTraceForRetry(generation); // retry later or serve score without trace
  } else throw e;
}

Prevention

Check setCachedJson's return contract and log the underlying Redis error, not just falsy
Add bounded retries with backoff inside the persist path
Monitor Redis write health and quotas in production
Decide explicitly whether traces are required or best-effort, and handle accordingly

When it happens

Trigger: setCachedJson returns a falsy result for the trace key — typically because the Redis/Upstash write failed (HTTP error, quota, timeout) or the cache helper signals failure by returning null/false instead of throwing.

Common situations: Upstash outage or rate limiting during score generation; expired/invalid Redis credentials in the deployment; key too large for the plan or write quota exhausted; cache helper contract change making it return null on transient errors.

Related errors

AI-assisted analysis of koala73/worldmonitor@9361220cc0 (2026-09-01). Data as JSON: /api/errors/4f6091b8a1f33e3c. Report an issue: GitHub.