What causes "IMD_RESPONSE_TOO_LARGE:${size}" in koala73/worldmonitor?

Increase maxBytes in the fetchApprovedImdJson call options if the payload legitimately grew.

IMD_RESPONSE_TOO_LARGE:${size}

Error message

IMD_RESPONSE_TOO_LARGE:${size}

What it means

readBoundedJsonResponse() in scripts/lib/imd-cyclone-marine.mjs streams the IMD HTTP response body and enforces a byte cap (default IMD_MAX_BYTES). As soon as the accumulated size exceeds maxBytes, it aborts the read and throws IMD_RESPONSE_TOO_LARGE:<bytes> so an oversized or malicious payload can never be fully buffered or parsed. The thrown message carries the byte count at which the limit was crossed.

Solutions

Increase maxBytes in the fetchApprovedImdJson call options if the payload legitimately grew.
Log the byte count from the error message and inspect what the endpoint actually returns (curl with -o and check size) — often it is an HTML error page, not JSON.
Check proxy/CDN configuration (compression, caching) that may inflate the response, and fix upstream instead.
If the dataset is genuinely too large, request a narrower product/endpoint from IMD rather than raising the cap blindly.

Example fix

// before
const data = await fetchApprovedImdJson(url, { maxBytes: 64 * 1024 });
// after
const data = await fetchApprovedImdJson(url, { maxBytes: 512 * 1024 });

Defensive patterns

Strategy: try-catch

Validate before calling

const HEADROOM = 2;
if (typeof maxBytes !== 'number' || maxBytes <= 0) {
  throw new Error(`maxBytes must be a positive number, got ${maxBytes}`);
}
// Optionally pre-check with a HEAD request:
const head = await fetch(url, { method: 'HEAD' });
const len = Number(head.headers.get('content-length'));
if (len && len > maxBytes) console.warn(`Expected response ${len}B exceeds maxBytes ${maxBytes}`);

Try / catch

try {
  const data = await fetchApprovedImdJson(url, { maxBytes: IMD_MAX_BYTES });
} catch (err) {
  if (String(err.message).startsWith('IMD_RESPONSE_TOO_LARGE')) {
    const bytes = Number(err.message.split(':')[1]);
    console.error(`IMD response exceeded limit at ${bytes} bytes; skipping or raising cap`);
    return null; // degrade gracefully, keep last good snapshot
  }
  throw err;
}

Prevention

Keep the last successfully parsed snapshot and serve stale data when this error fires, instead of crashing the worker.
Set maxBytes deliberately (documented constant), not by copying defaults blindly.
Alert on this error — repeated occurrences usually mean the upstream endpoint changed shape.
Monitor content-length on HEAD requests to catch payload growth before the limit is hit.

When it happens

Trigger: Calling fetchApprovedImdJson(url) (directly or via a proxy fetch) where the response body from api.maalaimaatham/IMD host exceeds maxBytes while streaming, or where response.body lacks getReader() and response.text() yields more than maxBytes bytes.

Common situations: The IMD endpoint returns an unexpectedly huge bulletin or HTML error page instead of JSON; a caller passes a very small custom maxBytes to fetchApprovedImdJson; a misconfigured proxy returns a large multi-MB response; a compromised/upstream-changed endpoint streams unbounded data (exactly what the guard protects against).

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of koala73/worldmonitor@7d06c8633d (2026-09-15). Data as JSON: /api/errors/c5cd39083ed2d0aa. Report an issue: GitHub.

Appendix: source

Thrown at scripts/lib/imd-cyclone-marine.mjs:740

    marineBulletins: marineBulletinsFromSnapshot(snapshot),
  };
}

async function readBoundedJsonResponse(response, maxBytes = IMD_MAX_BYTES) {
  const chunks = [];
  let size = 0;
  if (!response.body || typeof response.body.getReader !== 'function') {
    const text = await response.text();
    const bytes = Buffer.byteLength(text, 'utf8');
    if (bytes > maxBytes) throw new Error(`IMD_RESPONSE_TOO_LARGE:${bytes}`);
    return JSON.parse(text);
  }
  const reader = response.body.getReader();
  for (;;) {
    const { done, value } = await reader.read();
    if (done) break;
    size += value.byteLength;
    if (size > maxBytes) throw new Error(`IMD_RESPONSE_TOO_LARGE:${size}`);
    chunks.push(value);
  }
  return JSON.parse(new TextDecoder().decode(Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)))));
}

export async function fetchApprovedImdJson(url, {
  fetchFn = globalThis.fetch,
  userAgent = CHROME_UA,
  maxBytes = IMD_MAX_BYTES,
  timeoutMs = IMD_TIMEOUT_MS,
  apiKey = null,
  apiKeyHeader = 'X-API-Key',
  apiToken = null,
} = {}) {
  if (!isAllowedImdHost(url)) throw new Error('UNTRUSTED_SOURCE_HOST');
  const headers = { Accept: 'application/json', 'User-Agent': userAgent };
  if (apiKey) headers[apiKeyHeader] = apiKey;
  if (apiToken) headers.Authorization = `Bearer ${apiToken}`;

View on GitHub (pinned to 7d06c8633d)