{"record":{"id":"f14d51d5fe1c676f","repo":"koala73/worldmonitor","slug":"webhook-index-could-not-be-read-or-cleaned","errorCode":null,"errorMessage":"Webhook index could not be read or cleaned","messagePattern":"Webhook index could not be read or cleaned","errorType":"http","errorClass":"ApiError","httpStatus":503,"severity":"error","filePath":"server/worldmonitor/shipping/v2/webhook-owner-index.ts","lineNumber":9,"sourceCode":"import { ApiError } from '../../../../src/generated/server/worldmonitor/shipping/v2/service_server';\nimport { runRedisPipeline } from '../../../_shared/redis';\nimport { ownerIndexKey, webhookKey, WEBHOOK_TTL } from './webhook-shared';\n\nconst SWEEP_BATCH_SIZE = 100;\nconst REMOVE_EXPIRED_MEMBER = \"if redis.call('EXISTS', KEYS[2]) == 0 then return redis.call('SREM', KEYS[1], ARGV[1]) else return 0 end\";\n\nfunction unavailable(): never {\n  throw new ApiError(503, 'Webhook index could not be read or cleaned', '');\n}\n\nfunction stringArray(value: unknown): value is string[] {\n  return Array.isArray(value) && value.every((item) => typeof item === 'string');\n}\n\nexport async function pruneOwnerWebhookIndex(ownerTag: string): Promise<void> {\n  const ownerKey = ownerIndexKey(ownerTag);\n  const sweepKey = `${ownerKey}:sweep`;\n  const stateResult = await runRedisPipeline([['GET', sweepKey]]);\n  if (!stateResult[0] || stateResult[0].error) unavailable();\n  let state = { cursor: '0', offset: 0 };\n  if (stateResult[0].result !== null) {\n    if (typeof stateResult[0].result !== 'string') unavailable();\n    try {\n      const parsed = JSON.parse(stateResult[0].result);\n      if (!parsed || typeof parsed.cursor !== 'string' || !/^\\d+$/.test(parsed.cursor)\n        || !Number.isSafeInteger(parsed.offset) || parsed.offset < 0) unavailable();","sourceCodeStart":1,"sourceCodeEnd":27,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/server/worldmonitor/shipping/v2/webhook-owner-index.ts#L1-L27","documentation":"The webhook owner-index module keeps a Redis set of webhook keys per owner and prunes expired members with a Lua script. Its `unavailable()` helper throws HTTP 503 'Webhook index could not be read or cleaned' whenever a required Redis read or cleanup operation fails or returns an unexpected type (non-string-array), and it is raised by pruneOwnerWebhookIndex and readOwnerWebhooks.","triggerScenarios":"Calling register-webhook or list-webhooks for an owner while the owner-index Redis read/SREM scan fails: Redis down, timeout, wrong type at the index key, or the Lua sweep returning a malformed result.","commonSituations":"Redis outage or degraded latency; a key of the wrong type at the owner-index path (e.g. written by an older schema); transient network failures between worker and Redis; Upstash rate limits.","solutions":["Retry the operation after a brief backoff; index cleanup is best-effort and transient failures usually clear","Check Redis connectivity and the type of the owner-index key (should be a set); delete/repair mistyped keys","Confirm no older deployment wrote a conflicting schema to webhook-index keys","If persistent, verify Redis quotas (Upstash limits) and error dashboards; the server cannot list/register webhooks until reads succeed"],"exampleFix":"// before\nconst hooks = await client.listWebhooks({ ownerId }); // 503 if index read fails\n// after\ntry {\n  hooks = await client.listWebhooks({ ownerId });\n} catch (e) {\n  if (e.status === 503) {\n    await sleep(1000);\n    return client.listWebhooks({ ownerId }); // single retry for transient Redis issues\n  }\n  throw e;\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return await client.listWebhooks({ ownerId });\n} catch (e) {\n  if (e.status === 503 && /index could not be read/.test(e.message)) {\n    await sleep(1000);\n    return client.listWebhooks({ ownerId });\n  }\n  throw e;\n}","preventionTips":["Add bounded retries for 503 storage errors on read paths","Monitor Redis health and key types for webhook index keys","Avoid schema-breaking manual writes to Redis webhook keys"],"tags":["redis","webhook","http-503","storage"],"backgroundTag":"database-query-failed","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}