koala73/worldmonitor · warning · ApiError

Scenario queue is at capacity, please try again later

Error message

Scenario queue is at capacity, please try again later

What it means

Thrown as HTTP 429 by runScenario when the Redis list 'scenario-queue:pending' (read unprefixed via LLEN because the worker reads the raw key) holds more than MAX_QUEUE_DEPTH (100) entries. It is deliberate queue-depth backpressure so a slow or stopped scenario worker cannot accumulate unbounded pending jobs. The job was never created; no resources were consumed on the server.

Source

Thrown at server/worldmonitor/scenario/v1/run-scenario.ts:51

  const scenarioId = (req.scenarioId ?? '').trim();
  if (!scenarioId) {
    throw new ValidationError([{ field: 'scenarioId', description: 'scenarioId is required' }]);
  }
  if (!getScenarioTemplate(scenarioId)) {
    throw new ValidationError([{ field: 'scenarioId', description: `Unknown scenario: ${scenarioId}` }]);
  }

  const iso2 = req.iso2 ? req.iso2.trim() : '';
  if (iso2 && !/^[A-Z]{2}$/.test(iso2)) {
    throw new ValidationError([{ field: 'iso2', description: 'iso2 must be a 2-letter uppercase country code' }]);
  }

  // Queue-depth backpressure. Raw key: worker reads it unprefixed, so we must too.
  const [depthEntry] = await runRedisPipeline([['LLEN', QUEUE_KEY]], true);
  const depth = typeof depthEntry?.result === 'number' ? depthEntry.result : 0;
  if (depth > MAX_QUEUE_DEPTH) {
    throw new ApiError(429, 'Scenario queue is at capacity, please try again later', '');
  }

  const jobId = generateJobId();
  const payload = JSON.stringify({
    jobId,
    scenarioId,
    iso2: iso2 || null,
    enqueuedAt: Date.now(),
  });

  // Upstash RPUSH returns the new list length; helper returns [] on transport
  // failure. Either no entry or a non-numeric result means the enqueue never
  // landed — surface as 502 so the caller retries.
  const [pushEntry] = await runRedisPipeline([['RPUSH', QUEUE_KEY, payload]], true);
  if (!pushEntry || typeof pushEntry.result !== 'number') {
    throw new ApiError(502, 'Failed to enqueue scenario job', '');
  }

View on GitHub (pinned to eeab0a219f)

Solutions

1. Check the scenario worker is running and draining 'scenario-queue:pending' (LLEN should decrease over time); restart it if stopped
2. Retry the RunScenario call with exponential backoff and jitter, honoring the 429
3. If the backlog is stale residue from an old incident, have an operator inspect and trim scenario-queue:pending in Redis
4. If depth 100 is genuinely too low for sustained traffic, raise MAX_QUEUE_DEPTH in server/worldmonitor/scenario/v1/run-scenario.ts:16 and scale the worker

Example fix

// before
const res = await client.runScenario({ scenarioId: 'panama-closure' }); // 429 at capacity
// after
async function runWithBackoff(req, attempts = 5) {
  for (let i = 0; i < attempts; i++) {
    try { return await client.runScenario(req); }
    catch (e) {
      if (e?.status === 429 && i < attempts - 1) {
        await new Promise(r => setTimeout(r, 1000 * 2 ** i));
        continue;
      }
      throw e;
    }
  }
}

try { await runScenario(req); } catch (e) { if (e?.status === 429) { await sleep(backoffMs(jitter)); return retry(req); } throw e; } — 429 means the job was never enqueued, so plain bounded retry with exponential backoff is safe.

When it happens

Trigger: POST RunScenario after the PRO gate and scenarioId/iso2 validation pass while LLEN scenario-queue:pending > 100. Typical causes: the scenario worker process is stopped or wedged, a burst of >100 concurrent runs outruns the worker drain rate, or a stale backlog remains in Redis after a worker outage.

Common situations: Scenario worker crashed or not deployed (the queue only drains via the worker); load test or batch script firing many RunScenario calls; local development with the API running but no worker; abandoned jobs from a previous incident still sitting in the list.

Related errors

• Failed to enqueue scenario job
• Exa ${endpoint.slice(1)} failed HTTP ${response.status}: ${d
• Revoke failed (HTTP ${resp.status}).
• Firecrawl extract error: ${data.error ?? 'unknown'}
• P0 scrape failed: HTTP ${resp.status}

AI-assisted analysis of koala73/worldmonitor@eeab0a219f (2026-08-21). Data as JSON: /api/errors/4364aac846e346ad. Report an issue: GitHub.