{"record":{"id":"7207f37ad1f40c29","repo":"koala73/worldmonitor","slug":"imd-response-too-large-bytes","errorCode":null,"errorMessage":"IMD_RESPONSE_TOO_LARGE:${bytes}","messagePattern":"IMD_RESPONSE_TOO_LARGE:(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scripts/lib/imd-cyclone-marine.mjs","lineNumber":732,"sourceCode":"  });\n}\n\nfunction decorateSnapshotSurfaces(snapshot) {\n  return {\n    ...snapshot,\n    cycloneEvents: cycloneEventsFromSnapshot(snapshot),\n    portAlerts: weatherAlertsFromSnapshot(snapshot),\n    marineBulletins: marineBulletinsFromSnapshot(snapshot),\n  };\n}\n\nasync function readBoundedJsonResponse(response, maxBytes = IMD_MAX_BYTES) {\n  const chunks = [];\n  let size = 0;\n  if (!response.body || typeof response.body.getReader !== 'function') {\n    const text = await response.text();\n    const bytes = Buffer.byteLength(text, 'utf8');\n    if (bytes > maxBytes) throw new Error(`IMD_RESPONSE_TOO_LARGE:${bytes}`);\n    return JSON.parse(text);\n  }\n  const reader = response.body.getReader();\n  for (;;) {\n    const { done, value } = await reader.read();\n    if (done) break;\n    size += value.byteLength;\n    if (size > maxBytes) throw new Error(`IMD_RESPONSE_TOO_LARGE:${size}`);\n    chunks.push(value);\n  }\n  return JSON.parse(new TextDecoder().decode(Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)))));\n}\n\nexport async function fetchApprovedImdJson(url, {\n  fetchFn = globalThis.fetch,\n  userAgent = CHROME_UA,\n  maxBytes = IMD_MAX_BYTES,\n  timeoutMs = IMD_TIMEOUT_MS,","sourceCodeStart":714,"sourceCodeEnd":750,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/lib/imd-cyclone-marine.mjs#L714-L750","documentation":"readBoundedJsonResponse in scripts/lib/imd-cyclone-marine.mjs caps IMD (India Meteorological Department) JSON responses at maxBytes (default IMD_MAX_BYTES). When the accumulated body — either from response.text() on a non-streaming body or from summing stream chunks — exceeds the cap, it throws IMD_RESPONSE_TOO_LARGE:<byteCount>, embedding the actual size in the message so callers can see the overage.","triggerScenarios":"An IMD endpoint (cyclone, marine bulletin, or forecast product JSON) returns more bytes than IMD_MAX_BYTES; a product feed returns the full archive instead of a single page; the caller passed a custom maxBytes below the typical payload size in fetchApprovedImdJson options.","commonSituations":"IMD API silently returns an unpaginated bulk response during severe-weather events; proxy or gateway decompression inflates payload size; developer lowers maxBytes for testing and forgets to restore it; new IMD product added whose responses exceed the default cap.","solutions":["Read the byte count in the error message (e.g. IMD_RESPONSE_TOO_LARGE:5242880) and compare it with IMD_MAX_BYTES / the maxBytes option to size the overage.","If the product's responses legitimately exceed the cap, raise maxBytes (or IMD_MAX_BYTES) to a justified value when calling fetchApprovedImdJson.","Check whether the endpoint supports pagination/filtering (smaller product windows) and request that instead of the bulk payload.","Verify no proxy/gateway is inflating responses (e.g. concatenating payloads or disabling compression) and that you are hitting the intended product URL."],"exampleFix":"// before: default cap rejects a legitimately larger new IMD product\nconst data = await fetchApprovedImdJson(url);\n\n// after: raise the cap for this product with a justified limit\nconst data = await fetchApprovedImdJson(url, { maxBytes: 10 * 1024 * 1024 });","handlingStrategy":"validation","validationCode":"// Pre-flight: check content-length before reading when available\nconst len = Number(response.headers.get('content-length'));\nif (Number.isFinite(len) && len > maxBytes) {\n  throw new Error(`IMD_RESPONSE_TOO_LARGE:${len}`); // avoid reading at all\n}","typeGuard":null,"tryCatchPattern":"try {\n  const data = await fetchApprovedImdJson(url, { maxBytes });\n  return data;\n} catch (error) {\n  if (String(error?.message).startsWith('IMD_RESPONSE_TOO_LARGE:')) {\n    const bytes = Number(error.message.split(':')[1]);\n    logger.warn({ url, bytes, maxBytes }, 'IMD response exceeded cap');\n    return null; // skip this product\n  }\n  throw error;\n}","preventionTips":["Check content-length headers before reading large IMD feeds.","Size maxBytes from observed production payload percentiles, not guesses.","Prefer paginated/filtered IMD product endpoints over bulk downloads.","Add monitoring that records response sizes per IMD product so cap growth is data-driven."],"tags":["payload-too-large","response-limit","imd","json"],"backgroundTag":"payload-too-large","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}