koala73/worldmonitor · error · Error

Failed to persist resilience score generation for ${generati

Error message

Failed to persist resilience score generation for ${generation.trace.countryCode}

What it means

This error is thrown in server/worldmonitor/resilience/v1/_shared.ts when the resilience score generation payload could not be written to the shared cache via setCachedJson(). The library treats a cache write failure as fatal for score generation because subsequent reads rely on the cached payload being present under scoreCacheKey(countryCode); returning a score without persisting it would break trace-consistency guarantees. It wraps the underlying cache backend failure (Redis/Upstash outage, TTL/storage error) into an explicit generation error.

Source

Thrown at server/worldmonitor/resilience/v1/_shared.ts:1105

    scoreTraceCacheKey(generation.trace.countryCode, generation.trace.generationId),
    generation.trace,
    RESILIENCE_SCORE_TRACE_CACHE_TTL_SECONDS,
  );
  if (!persisted) {
    throw new Error(`Failed to persist resilience trace generation for ${generation.trace.countryCode}`);
  }
}

async function persistFullScoreGeneration(generation: ResilienceScoreGeneration): Promise<CachedScorePayload> {
  await persistGenerationTrace(generation);
  const cachedPayload = cachedPayloadForGeneration(generation);
  const scorePersisted = await setCachedJson(
    scoreCacheKey(generation.trace.countryCode),
    cachedPayload,
    RESILIENCE_SCORE_CACHE_TTL_SECONDS,
  );
  if (!scorePersisted) {
    throw new Error(`Failed to persist resilience score generation for ${generation.trace.countryCode}`);
  }
  return cachedPayload;
}

async function toPublicCachedScore(
  countryCode: string,
  cached: CachedScorePayload,
): Promise<GetResilienceScoreResponse> {
  let payload = stripCacheMeta(cached);
  const scoreInterval = await readScoreInterval(countryCode);
  if (scoreInterval) payload = { ...payload, scoreInterval };

  // The cache stores the v2 superset. Gate the public shape at serve time so a
  // schema rollback takes effect without waiting for the score TTL.
  if (!RESILIENCE_SCHEMA_V2_ENABLED) {
    payload.pillars = [];
    payload.schemaVersion = '1.0';
  }

View on GitHub (pinned to 9361220cc0)

Solutions

Check the cache backend (Redis/Upstash) is reachable and credentials are valid; retry once the cache recovers since this is a transient infrastructure failure.
Verify RESILIENCE_SCORE_CACHE_TTL_SECONDS and cache env configuration (keys/endpoints) are set correctly for the deployment.
Inspect cachedPayload size; if it exceeds the backend limit, trim the trace sidecar or raise the backend's value-size limit.
Wrap the score generation call in a retry with backoff, and surface a degraded response instead of failing the request while the cache is unavailable.

Example fix

// before
const scorePersisted = await setCachedJson(scoreCacheKey(generation.trace.countryCode), cachedPayload, RESILIENCE_SCORE_CACHE_TTL_SECONDS);
if (!scorePersisted) {
  throw new Error(`Failed to persist resilience score generation for ${generation.trace.countryCode}`);
}
// after
const scorePersisted = await setCachedJson(scoreCacheKey(generation.trace.countryCode), cachedPayload, RESILIENCE_SCORE_CACHE_TTL_SECONDS);
if (!scorePersisted) {
  console.warn(`score cache write failed for ${generation.trace.countryCode}; serving uncacheable response`);
  return cachedPayload; // degrade gracefully instead of throwing
}

Defensive patterns

Strategy: try-catch

Validate before calling

// preflight cache reachability
const cacheOk = await checkCacheHealth(); // e.g. PING the Redis client
if (!cacheOk) console.warn('cache unavailable; score generation may fail');

Try / catch

try {
  const score = await ensureResilienceScoreCached(cc);
} catch (err) {
  if (err instanceof Error && err.message.startsWith('Failed to persist resilience score generation')) {
    // transient cache outage: retry with backoff or serve degraded response
    await sleep(RETRY_MS);
    return serveDegraded(cc);
  }
  throw err;
}

Prevention

Monitor cache backend health/latency and alert before it saturates
Keep RESILIENCE_SCORE_CACHE_TTL_SECONDS and cache env config in one validated module
Cap payload size (trim trace sidecars) under the backend value limit
Retry transient SET failures with backoff inside the cache helper

When it happens

Trigger: Calling ensureResilienceScoreCached (or any path that builds a fresh score generation) when setCachedJson(scoreCacheKey(...), cachedPayload, RESILIENCE_SCORE_CACHE_TTL_SECONDS) returns false — i.e., the cache client returned a null/error result on SET, typically during a Redis/Upstash outage or when the cached payload exceeds backend size limits.

Common situations: 1) Cache service (Upstash Redis) temporarily down or rate-limited while the score cache is cold; 2) serialized cachedPayload exceeds the cache backend's max value size after schema-v2 payloads grew; 3) misconfigured or expired cache credentials in the server environment; 4) transient network partition between the Railway server and the cache.

Related errors

AI-assisted analysis of koala73/worldmonitor@9361220cc0 (2026-09-01). Data as JSON: /api/errors/d476212cec199fd4. Report an issue: GitHub.