{"record":{"id":"c5cd39083ed2d0aa","repo":"koala73/worldmonitor","slug":"imd-response-too-large-size","errorCode":null,"errorMessage":"IMD_RESPONSE_TOO_LARGE:${size}","messagePattern":"IMD_RESPONSE_TOO_LARGE:(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scripts/lib/imd-cyclone-marine.mjs","lineNumber":740,"sourceCode":"    marineBulletins: marineBulletinsFromSnapshot(snapshot),\n  };\n}\n\nasync function readBoundedJsonResponse(response, maxBytes = IMD_MAX_BYTES) {\n  const chunks = [];\n  let size = 0;\n  if (!response.body || typeof response.body.getReader !== 'function') {\n    const text = await response.text();\n    const bytes = Buffer.byteLength(text, 'utf8');\n    if (bytes > maxBytes) throw new Error(`IMD_RESPONSE_TOO_LARGE:${bytes}`);\n    return JSON.parse(text);\n  }\n  const reader = response.body.getReader();\n  for (;;) {\n    const { done, value } = await reader.read();\n    if (done) break;\n    size += value.byteLength;\n    if (size > maxBytes) throw new Error(`IMD_RESPONSE_TOO_LARGE:${size}`);\n    chunks.push(value);\n  }\n  return JSON.parse(new TextDecoder().decode(Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)))));\n}\n\nexport async function fetchApprovedImdJson(url, {\n  fetchFn = globalThis.fetch,\n  userAgent = CHROME_UA,\n  maxBytes = IMD_MAX_BYTES,\n  timeoutMs = IMD_TIMEOUT_MS,\n  apiKey = null,\n  apiKeyHeader = 'X-API-Key',\n  apiToken = null,\n} = {}) {\n  if (!isAllowedImdHost(url)) throw new Error('UNTRUSTED_SOURCE_HOST');\n  const headers = { Accept: 'application/json', 'User-Agent': userAgent };\n  if (apiKey) headers[apiKeyHeader] = apiKey;\n  if (apiToken) headers.Authorization = `Bearer ${apiToken}`;","sourceCodeStart":722,"sourceCodeEnd":758,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/lib/imd-cyclone-marine.mjs#L722-L758","documentation":"readBoundedJsonResponse() in scripts/lib/imd-cyclone-marine.mjs streams the IMD HTTP response body and enforces a byte cap (default IMD_MAX_BYTES). As soon as the accumulated size exceeds maxBytes, it aborts the read and throws IMD_RESPONSE_TOO_LARGE:<bytes> so an oversized or malicious payload can never be fully buffered or parsed. The thrown message carries the byte count at which the limit was crossed.","triggerScenarios":"Calling fetchApprovedImdJson(url) (directly or via a proxy fetch) where the response body from api.maalaimaatham/IMD host exceeds maxBytes while streaming, or where response.body lacks getReader() and response.text() yields more than maxBytes bytes.","commonSituations":"The IMD endpoint returns an unexpectedly huge bulletin or HTML error page instead of JSON; a caller passes a very small custom maxBytes to fetchApprovedImdJson; a misconfigured proxy returns a large multi-MB response; a compromised/upstream-changed endpoint streams unbounded data (exactly what the guard protects against).","solutions":["Increase maxBytes in the fetchApprovedImdJson call options if the payload legitimately grew.","Log the byte count from the error message and inspect what the endpoint actually returns (curl with -o and check size) — often it is an HTML error page, not JSON.","Check proxy/CDN configuration (compression, caching) that may inflate the response, and fix upstream instead.","If the dataset is genuinely too large, request a narrower product/endpoint from IMD rather than raising the cap blindly."],"exampleFix":"// before\nconst data = await fetchApprovedImdJson(url, { maxBytes: 64 * 1024 });\n// after\nconst data = await fetchApprovedImdJson(url, { maxBytes: 512 * 1024 });","handlingStrategy":"try-catch","validationCode":"const HEADROOM = 2;\nif (typeof maxBytes !== 'number' || maxBytes <= 0) {\n  throw new Error(`maxBytes must be a positive number, got ${maxBytes}`);\n}\n// Optionally pre-check with a HEAD request:\nconst head = await fetch(url, { method: 'HEAD' });\nconst len = Number(head.headers.get('content-length'));\nif (len && len > maxBytes) console.warn(`Expected response ${len}B exceeds maxBytes ${maxBytes}`);","typeGuard":null,"tryCatchPattern":"try {\n  const data = await fetchApprovedImdJson(url, { maxBytes: IMD_MAX_BYTES });\n} catch (err) {\n  if (String(err.message).startsWith('IMD_RESPONSE_TOO_LARGE')) {\n    const bytes = Number(err.message.split(':')[1]);\n    console.error(`IMD response exceeded limit at ${bytes} bytes; skipping or raising cap`);\n    return null; // degrade gracefully, keep last good snapshot\n  }\n  throw err;\n}","preventionTips":["Keep the last successfully parsed snapshot and serve stale data when this error fires, instead of crashing the worker.","Set maxBytes deliberately (documented constant), not by copying defaults blindly.","Alert on this error — repeated occurrences usually mean the upstream endpoint changed shape.","Monitor content-length on HEAD requests to catch payload growth before the limit is hit."],"tags":["network","payload-too-large","security","imd"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}