{"record":{"id":"c74d5d7c06e98cfe","repo":"koala73/worldmonitor","slug":"eccc-aggregate-too-large","errorCode":"ECCC_AGGREGATE_TOO_LARGE","errorMessage":"ECCC_AGGREGATE_TOO_LARGE","messagePattern":"ECCC_AGGREGATE_TOO_LARGE","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/_weather-alert-select.mjs","lineNumber":763,"sourceCode":"export async function fetchEcccAlertFeatures({\n  fetchFn = globalThis.fetch,\n  userAgent,\n  maxBytes = ECCC_MAX_BYTES,\n} = {}) {\n  const byteBudget = { remaining: ECCC_MAX_AGGREGATE_BYTES };\n  let pages = 0;\n  const seenIds = new Set();\n  const features = [];\n  const failures = [];\n  const failedStatuses = [];\n  // Sequential paging makes the shared resource limits deterministic.\n  for (const [index, status] of ECCC_LIVE_STATUSES.entries()) {\n    const statusFeatures = [];\n    let matched;\n    try {\n      do {\n        if (pages >= ECCC_MAX_PAGES) throw new Error('ECCC_PAGE_LIMIT');\n        if (byteBudget.remaining <= 0) throw new Error('ECCC_AGGREGATE_TOO_LARGE');\n        const url = new URL(ECCC_ALERTS_URLS[index]);\n        url.searchParams.set('offset', String(statusFeatures.length));\n        pages += 1;\n        const data = await fetchApprovedWeatherJson(url.toString(), {\n          allowedHosts: [ECCC_HOST], maxBytes, fetchFn, userAgent, byteBudget,\n        });\n        const page = requireAlertFeatures(data);\n        if (data.type !== 'FeatureCollection'\n          || !Number.isSafeInteger(data.numberMatched) || data.numberMatched < 0\n          || !Number.isSafeInteger(data.numberReturned) || data.numberReturned !== page.length\n          || page.length > ECCC_PAGE_SIZE) {\n          throw new Error('ECCC_MALFORMED_PAGE');\n        }\n        if (matched !== undefined && data.numberMatched !== matched) throw new Error('ECCC_COUNT_DRIFT');\n        matched = data.numberMatched;\n        if (statusFeatures.length + page.length > matched\n          || (page.length === 0 && statusFeatures.length < matched)) {\n          throw new Error('ECCC_PAGE_PROGRESS');","sourceCodeStart":745,"sourceCodeEnd":781,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/_weather-alert-select.mjs#L745-L781","documentation":"All ECCC alert pages share one aggregate byte budget of ECCC_MAX_AGGREGATE_BYTES = 8 MiB across the 'issued' and 'continued' collections (each single page is also capped at ECCC_MAX_BYTES = 4 MiB). The loop checks `byteBudget.remaining <= 0` before each page and throws Error('ECCC_AGGREGATE_TOO_LARGE') to prevent the combined national alert payload from growing unbounded. readResponseLimited() also throws this when streaming bytes push the shared budget negative.","triggerScenarios":"Fetching a page in scripts/_weather-alert-select.mjs:763 when the running total of all previously read pages has exhausted the 8 MiB shared budget (byteBudget.remaining <= 0), or inside readResponseLimited when cumulative streamed bytes drive byteBudget.remaining below 0.","commonSituations":"An unusually active severe-weather day where issued+continued alerts together exceed 8 MiB of JSON; an upstream API change that inflates each feature's payload size; a misconfigured caller raising maxBytes so individual pages no longer trip RESPONSE_TOO_LARGE but the aggregate does; a stub fetch in tests returning very large bodies for many pages.","solutions":["Inspect failureDetail from the aggregate error to see whether 'issued' or 'continued' (or both) exhausted the budget; a single status failing yields a partial result, not a throw.","If real alert volume legitimately exceeds 8 MiB, raise ECCC_MAX_AGGREGATE_BYTES in scripts/_weather-alert-select.mjs:48, keeping ECCC_MAX_BYTES per page within the deliberate 2–4 MiB ceiling.","Check whether the ECCC API started returning extra fields or verbose geometries that bloat each feature; consider trimming or compacting the payload upstream.","Confirm no earlier page consumed budget twice (byteBudget is decremented inside readResponseLimited per chunk; don't decrement again at call sites)."],"exampleFix":"// before\nconst ECCC_MAX_AGGREGATE_BYTES = 8 * 1024 * 1024;\n// after (national alert set legitimately grew)\nconst ECCC_MAX_AGGREGATE_BYTES = 12 * 1024 * 1024;","handlingStrategy":"try-catch","validationCode":"// Estimate aggregate size before running the full fetch\nconst headIssued = await fetchFn(urlIssued + '&limit=1');\nconst headCont = await fetchFn(urlContinued + '&limit=1');\nconst matched = (await headIssued.json()).numberMatched + (await headCont.json()).numberMatched;\n// ~3KB average per feature; warn if near the 8 MiB aggregate budget\nif (matched * 3000 > 8 * 1024 * 1024) console.warn('ECCC aggregate byte budget at risk');","typeGuard":null,"tryCatchPattern":"try {\n  const result = await fetchEcccAlertFeatures({ fetchFn, userAgent });\n} catch (err) {\n  if (String(err.message).includes('ECCC_AGGREGATE_TOO_LARGE')) {\n    reportHealth('eccc', { ok: false, reason: 'aggregate-too-large', detail: err.message });\n  } else throw err;\n}","preventionTips":["Track typical per-feature payload size; revisit ECCC_MAX_AGGREGATE_BYTES if the API adds fields.","Keep maxBytes (per page, 4 MiB) coherent with the 8 MiB aggregate budget — raising one without the other hides failures.","Never decrement byteBudget manually at call sites; only readResponseLimited owns it.","Check failureDetail to identify which status exhausted the budget before tuning constants."],"tags":["byte-budget","payload-too-large","pagination","api"],"backgroundTag":"payload-too-large","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}