{"record":{"id":"4364aac846e346ad","repo":"koala73/worldmonitor","slug":"scenario-queue-is-at-capacity-please-try-again-la","errorCode":null,"errorMessage":"Scenario queue is at capacity, please try again later","messagePattern":"Scenario queue is at capacity, please try again later","errorType":"http","errorClass":"ApiError","httpStatus":429,"severity":"warning","filePath":"server/worldmonitor/scenario/v1/run-scenario.ts","lineNumber":51,"sourceCode":"\n  const scenarioId = (req.scenarioId ?? '').trim();\n  if (!scenarioId) {\n    throw new ValidationError([{ field: 'scenarioId', description: 'scenarioId is required' }]);\n  }\n  if (!getScenarioTemplate(scenarioId)) {\n    throw new ValidationError([{ field: 'scenarioId', description: `Unknown scenario: ${scenarioId}` }]);\n  }\n\n  const iso2 = req.iso2 ? req.iso2.trim() : '';\n  if (iso2 && !/^[A-Z]{2}$/.test(iso2)) {\n    throw new ValidationError([{ field: 'iso2', description: 'iso2 must be a 2-letter uppercase country code' }]);\n  }\n\n  // Queue-depth backpressure. Raw key: worker reads it unprefixed, so we must too.\n  const [depthEntry] = await runRedisPipeline([['LLEN', QUEUE_KEY]], true);\n  const depth = typeof depthEntry?.result === 'number' ? depthEntry.result : 0;\n  if (depth > MAX_QUEUE_DEPTH) {\n    throw new ApiError(429, 'Scenario queue is at capacity, please try again later', '');\n  }\n\n  const jobId = generateJobId();\n  const payload = JSON.stringify({\n    jobId,\n    scenarioId,\n    iso2: iso2 || null,\n    enqueuedAt: Date.now(),\n  });\n\n  // Upstash RPUSH returns the new list length; helper returns [] on transport\n  // failure. Either no entry or a non-numeric result means the enqueue never\n  // landed — surface as 502 so the caller retries.\n  const [pushEntry] = await runRedisPipeline([['RPUSH', QUEUE_KEY, payload]], true);\n  if (!pushEntry || typeof pushEntry.result !== 'number') {\n    throw new ApiError(502, 'Failed to enqueue scenario job', '');\n  }\n","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/koala73/worldmonitor/blob/eeab0a219fce0f02a00603b532dbae9041b934ac/server/worldmonitor/scenario/v1/run-scenario.ts#L33-L69","documentation":"Thrown as HTTP 429 by runScenario when the Redis list 'scenario-queue:pending' (read unprefixed via LLEN because the worker reads the raw key) holds more than MAX_QUEUE_DEPTH (100) entries. It is deliberate queue-depth backpressure so a slow or stopped scenario worker cannot accumulate unbounded pending jobs. The job was never created; no resources were consumed on the server.","triggerScenarios":"POST RunScenario after the PRO gate and scenarioId/iso2 validation pass while LLEN scenario-queue:pending > 100. Typical causes: the scenario worker process is stopped or wedged, a burst of >100 concurrent runs outruns the worker drain rate, or a stale backlog remains in Redis after a worker outage.","commonSituations":"Scenario worker crashed or not deployed (the queue only drains via the worker); load test or batch script firing many RunScenario calls; local development with the API running but no worker; abandoned jobs from a previous incident still sitting in the list.","solutions":["Check the scenario worker is running and draining 'scenario-queue:pending' (LLEN should decrease over time); restart it if stopped","Retry the RunScenario call with exponential backoff and jitter, honoring the 429","If the backlog is stale residue from an old incident, have an operator inspect and trim scenario-queue:pending in Redis","If depth 100 is genuinely too low for sustained traffic, raise MAX_QUEUE_DEPTH in server/worldmonitor/scenario/v1/run-scenario.ts:16 and scale the worker"],"exampleFix":"// before\nconst res = await client.runScenario({ scenarioId: 'panama-closure' }); // 429 at capacity\n// after\nasync function runWithBackoff(req, attempts = 5) {\n  for (let i = 0; i < attempts; i++) {\n    try { return await client.runScenario(req); }\n    catch (e) {\n      if (e?.status === 429 && i < attempts - 1) {\n        await new Promise(r => setTimeout(r, 1000 * 2 ** i));\n        continue;\n      }\n      throw e;\n    }\n  }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { await runScenario(req); } catch (e) { if (e?.status === 429) { await sleep(backoffMs(jitter)); return retry(req); } throw e; } — 429 means the job was never enqueued, so plain bounded retry with exponential backoff is safe.","preventionTips":["Treat 429 from scenario run as transient: back off and retry, never tight-loop","Monitor scenario-queue:pending depth and alert before it approaches 100","Keep the scenario worker healthy — sustained 429s almost always mean the worker, not the API, is the bottleneck"],"tags":["rate-limit","backpressure","redis","queue","http-429","scenario"],"backgroundTag":"http-429-rate-limited","analyzedSha":"eeab0a219fce0f02a00603b532dbae9041b934ac","analyzedAt":"2026-08-21T16:51:25.751Z","contentChangedAt":"2026-08-21T16:51:25.751Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}