{"record":{"id":"cd305a77cdf84f27","repo":"koala73/worldmonitor","slug":"response-too-large","errorCode":"RESPONSE_TOO_LARGE","errorMessage":"RESPONSE_TOO_LARGE","messagePattern":"RESPONSE_TOO_LARGE","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scripts/cross-strait-activity/adapters.mjs","lineNumber":1734,"sourceCode":"  return text;\n}\n\nasync function fetchMndViaProxy(input, init, proxyConfig, proxyRequestFn) {\n  const maxResponseBytes = CROSS_STRAIT_SOURCE_CONTRACTS.taiwanMnd.maxResponseBytes;\n  const result = await proxyRequestFn(String(input), proxyConfig, {\n    headers: init.headers,\n    maxResponseBytes,\n    timeoutMs: REQUEST_TIMEOUT_MS,\n    signal: init.signal,\n  });\n  if (!Number.isInteger(result?.status) || result.status < 200 || result.status > 599) {\n    throw new Error('MND_PROXY_RESPONSE_INVALID');\n  }\n  if (result.status >= 300) {\n    return new Response(null, { status: result.status });\n  }\n  if (!Buffer.isBuffer(result.buffer)) throw new Error('MND_PROXY_RESPONSE_INVALID');\n  if (result.buffer.byteLength > maxResponseBytes) throw new Error('RESPONSE_TOO_LARGE');\n  return new Response(result.status === 204 || result.status === 205 ? null : result.buffer, {\n    status: result.status,\n  });\n}\n\nfunction shouldProxyJapanModFailure(error) {\n  const code = errorCode(error);\n  if (code === 'SOURCE_ERROR' || code === 'TIMEOUT' || code === 'JMOD_INDEX_EMPTY') return true;\n  const status = Number(/^HTTP_(\\d{3})$/u.exec(code)?.[1]);\n  return status === 403\n    || status === 408\n    || status === 425\n    || status === 429\n    || status >= 500;\n}\n\nasync function fetchJapanModViaConfiguredProxy(input, init, {\n  proxyUrl,","sourceCodeStart":1716,"sourceCodeEnd":1752,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/cross-strait-activity/adapters.mjs#L1716-L1752","documentation":"fetchMndViaProxy enforces the Taiwan MND source contract's maxResponseBytes (CROSS_STRAIT_SOURCE_CONTRACTS.taiwanMnd.maxResponseBytes) on proxy-fetched bodies. Even with a valid status and Buffer, if result.buffer.byteLength exceeds the cap the body is rejected with RESPONSE_TOO_LARGE so an oversized or hostile response is never parsed downstream.","triggerScenarios":"The MND proxy returns a 2xx Buffer body whose byteLength exceeds taiwanMnd.maxResponseBytes — e.g. the proxy ignores the maxResponseBytes option passed in init, the endpoint unexpectedly returns a huge page/binary, or the configured cap was lowered below typical response sizes.","commonSituations":"Proxy implementation streams the whole body without honoring maxResponseBytes truncation; upstream MND site change dramatically inflates page size; misconfigured/undersized maxResponseBytes in the source contract; proxy returning a compressed or inflated payload larger than expected.","solutions":["Log result.buffer.byteLength and the contract's maxResponseBytes to see by how much the response exceeded the cap.","Make the proxy honor the maxResponseBytes option by truncating/aborting the download at the cap instead of buffering the entire body.","If the real MND responses have legitimately grown, raise taiwanMnd.maxResponseBytes in CROSS_STRAIT_SOURCE_CONTRACTS to a justified value.","Check whether the proxy is accidentally decompressing or re-fetching a much larger resource than intended (wrong URL, redirects to big assets)."],"exampleFix":"// before: proxy buffers whole body, ignoring cap\nconst buffer = Buffer.concat(chunks);\nreturn { status, buffer };\n\n// after: enforce the cap inside the proxy as well\nif (total > maxResponseBytes) {\n  await reader.cancel().catch(() => {});\n  throw new Error('RESPONSE_TOO_LARGE');\n}\nreturn { status, buffer: Buffer.concat(chunks, total) };","handlingStrategy":"validation","validationCode":"// Guard before returning from the proxy layer\nconst cap = CROSS_STRAIT_SOURCE_CONTRACTS.taiwanMnd.maxResponseBytes;\nif (buffer.byteLength > cap) throw new Error(`RESPONSE_TOO_LARGE:${buffer.byteLength}>${cap}`);","typeGuard":null,"tryCatchPattern":"try {\n  return await fetchMndViaProxy(input, init, proxyConfig, proxyRequestFn);\n} catch (error) {\n  if (error?.message === 'RESPONSE_TOO_LARGE') {\n    logger.warn({ url: String(input) }, 'MND proxy response exceeded maxResponseBytes');\n    return null; // treat as unavailable, do not parse\n  }\n  throw error;\n}","preventionTips":["Enforce the byte cap inside the proxy while streaming, aborting at maxResponseBytes instead of buffering everything.","Review taiwanMnd.maxResponseBytes against real production payload sizes periodically.","Alert on responses within ~90% of the cap so legitimate growth is caught before hard failure.","Confirm the proxy isn't decompressing or redirecting to unexpectedly large assets."],"tags":["payload-too-large","proxy","response-limit","taiwan-mnd"],"backgroundTag":"payload-too-large","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}