koala73/worldmonitor · error · ApiError

Webhook index could not be read or cleaned

Error message

Webhook index could not be read or cleaned

What it means

The webhook owner-index module keeps a Redis set of webhook keys per owner and prunes expired members with a Lua script. Its `unavailable()` helper throws HTTP 503 'Webhook index could not be read or cleaned' whenever a required Redis read or cleanup operation fails or returns an unexpected type (non-string-array), and it is raised by pruneOwnerWebhookIndex and readOwnerWebhooks.

Solutions

1. Retry the operation after a brief backoff; index cleanup is best-effort and transient failures usually clear
2. Check Redis connectivity and the type of the owner-index key (should be a set); delete/repair mistyped keys
3. Confirm no older deployment wrote a conflicting schema to webhook-index keys
4. If persistent, verify Redis quotas (Upstash limits) and error dashboards; the server cannot list/register webhooks until reads succeed

Example fix

// before
const hooks = await client.listWebhooks({ ownerId }); // 503 if index read fails
// after
try {
  hooks = await client.listWebhooks({ ownerId });
} catch (e) {
  if (e.status === 503) {
    await sleep(1000);
    return client.listWebhooks({ ownerId }); // single retry for transient Redis issues
  }
  throw e;
}

try {
  return await client.listWebhooks({ ownerId });
} catch (e) {
  if (e.status === 503 && /index could not be read/.test(e.message)) {
    await sleep(1000);
    return client.listWebhooks({ ownerId });
  }
  throw e;
}

When it happens

Trigger: Calling register-webhook or list-webhooks for an owner while the owner-index Redis read/SREM scan fails: Redis down, timeout, wrong type at the index key, or the Lua sweep returning a malformed result.

Common situations: Redis outage or degraded latency; a key of the wrong type at the owner-index path (e.g. written by an older schema); transient network failures between worker and Redis; Upstash rate limits.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• Webhook registration could not be confirmed
• Service temporarily unavailable
• Revoke service is temporarily unavailable. Try again in a mo
• Redis not configured
• Redis HTTP ${resp.status}

AI-assisted analysis of koala73/worldmonitor@7d06c8633d (2026-09-15). Data as JSON: /api/errors/f14d51d5fe1c676f. Report an issue: GitHub.

Appendix: source

Thrown at server/worldmonitor/shipping/v2/webhook-owner-index.ts:9

import { ApiError } from '../../../../src/generated/server/worldmonitor/shipping/v2/service_server';
import { runRedisPipeline } from '../../../_shared/redis';
import { ownerIndexKey, webhookKey, WEBHOOK_TTL } from './webhook-shared';

const SWEEP_BATCH_SIZE = 100;
const REMOVE_EXPIRED_MEMBER = "if redis.call('EXISTS', KEYS[2]) == 0 then return redis.call('SREM', KEYS[1], ARGV[1]) else return 0 end";

function unavailable(): never {
  throw new ApiError(503, 'Webhook index could not be read or cleaned', '');
}

function stringArray(value: unknown): value is string[] {
  return Array.isArray(value) && value.every((item) => typeof item === 'string');
}

export async function pruneOwnerWebhookIndex(ownerTag: string): Promise<void> {
  const ownerKey = ownerIndexKey(ownerTag);
  const sweepKey = `${ownerKey}:sweep`;
  const stateResult = await runRedisPipeline([['GET', sweepKey]]);
  if (!stateResult[0] || stateResult[0].error) unavailable();
  let state = { cursor: '0', offset: 0 };
  if (stateResult[0].result !== null) {
    if (typeof stateResult[0].result !== 'string') unavailable();
    try {
      const parsed = JSON.parse(stateResult[0].result);
      if (!parsed || typeof parsed.cursor !== 'string' || !/^\d+$/.test(parsed.cursor)
        || !Number.isSafeInteger(parsed.offset) || parsed.offset < 0) unavailable();

View on GitHub (pinned to 7d06c8633d)