{"record":{"id":"a12c4a370adbfe88","repo":"koala73/worldmonitor","slug":"eccc-page-limit","errorCode":"ECCC_PAGE_LIMIT","errorMessage":"ECCC_PAGE_LIMIT","messagePattern":"ECCC_PAGE_LIMIT","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/_weather-alert-select.mjs","lineNumber":762,"sourceCode":" */\nexport async function fetchEcccAlertFeatures({\n  fetchFn = globalThis.fetch,\n  userAgent,\n  maxBytes = ECCC_MAX_BYTES,\n} = {}) {\n  const byteBudget = { remaining: ECCC_MAX_AGGREGATE_BYTES };\n  let pages = 0;\n  const seenIds = new Set();\n  const features = [];\n  const failures = [];\n  const failedStatuses = [];\n  // Sequential paging makes the shared resource limits deterministic.\n  for (const [index, status] of ECCC_LIVE_STATUSES.entries()) {\n    const statusFeatures = [];\n    let matched;\n    try {\n      do {\n        if (pages >= ECCC_MAX_PAGES) throw new Error('ECCC_PAGE_LIMIT');\n        if (byteBudget.remaining <= 0) throw new Error('ECCC_AGGREGATE_TOO_LARGE');\n        const url = new URL(ECCC_ALERTS_URLS[index]);\n        url.searchParams.set('offset', String(statusFeatures.length));\n        pages += 1;\n        const data = await fetchApprovedWeatherJson(url.toString(), {\n          allowedHosts: [ECCC_HOST], maxBytes, fetchFn, userAgent, byteBudget,\n        });\n        const page = requireAlertFeatures(data);\n        if (data.type !== 'FeatureCollection'\n          || !Number.isSafeInteger(data.numberMatched) || data.numberMatched < 0\n          || !Number.isSafeInteger(data.numberReturned) || data.numberReturned !== page.length\n          || page.length > ECCC_PAGE_SIZE) {\n          throw new Error('ECCC_MALFORMED_PAGE');\n        }\n        if (matched !== undefined && data.numberMatched !== matched) throw new Error('ECCC_COUNT_DRIFT');\n        matched = data.numberMatched;\n        if (statusFeatures.length + page.length > matched\n          || (page.length === 0 && statusFeatures.length < matched)) {","sourceCodeStart":744,"sourceCodeEnd":780,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/_weather-alert-select.mjs#L744-L780","documentation":"fetchEcccAlertFeatures() pages the ECCC weather-alerts OGC API (api.weather.gc.ca) with a hard cap of ECCC_MAX_PAGES = 8 pages shared across both 'issued' and 'continued' status collections. Before each page request it checks `pages >= ECCC_MAX_PAGES` and throws Error('ECCC_PAGE_LIMIT') to stop an unbounded paging loop. Thrown when the paged collections legitimately need more than 8 pages of 250 features (2000 alerts), or when pagination makes no progress so the loop never terminates.","triggerScenarios":"The `do { ... } while (statusFeatures.length < matched)` loop in scripts/_weather-alert-select.mjs:762 requests more than 8 pages in total across the 'issued' and 'continued' collections — i.e. numberMatched exceeds 2000 features for a status, or the offset-based paging keeps returning features so the loop continues past the page budget.","commonSituations":"A severe weather day where Canada-wide issued+continued alerts exceed 2000; an upstream API regression where the `limit` parameter is ignored (server returns fewer than ECCC_PAGE_SIZE=250 per page, forcing more pages); a broken `matched` value that never satisfies the loop-exit condition, causing endless paging; tests using a fetch stub that always reports more results.","solutions":["Check the actual alert volume: query https://api.weather.gc.ca/collections/weather-alerts/items?f=json&status_en=issued&limit=1 and inspect numberMatched; if it approaches 2000, raise ECCC_MAX_PAGES in scripts/_weather-alert-select.mjs:49.","Verify the server honors limit=250 by checking data.numberReturned per page; if pages return few items, the ECCC API changed its paging behavior and ECCC_PAGE_SIZE/maxBytes tuning is needed.","If a single status trips the limit, note fetchEcccAlertFeatures degrades gracefully: only when BOTH statuses fail does it throw the aggregate error; inspect failureDetail in the thrown message to see which statuses failed and why.","For tests, ensure the mocked fetch respects offset/limit paging and returns numberMatched consistent with finite pages so the loop terminates within 8 iterations."],"exampleFix":"// before\nconst ECCC_MAX_PAGES = 8;\n// after (if alert volume legitimately exceeds 2000)\nconst ECCC_MAX_PAGES = 16;","handlingStrategy":"try-catch","validationCode":"// Pre-check alert volume before paging\nconst probe = await fetchFn('https://api.weather.gc.ca/collections/weather-alerts/items?f=json&status_en=issued&limit=1');\nconst probeJson = await probe.json();\nconst expectedPages = Math.ceil((probeJson.numberMatched ?? 0) / 250);\nif (expectedPages > 8) console.warn('ECCC_PAGE_LIMIT risk: needs', expectedPages, 'pages');","typeGuard":"function hasFinitePageEstimate(envelope) {\n  return typeof envelope === 'object' && envelope !== null\n    && Number.isSafeInteger(envelope.numberMatched)\n    && Math.ceil(envelope.numberMatched / 250) <= 8;\n}","tryCatchPattern":"try {\n  const result = await fetchEcccAlertFeatures({ fetchFn, userAgent });\n} catch (err) {\n  if (String(err.message).includes('ECCC_PAGE_LIMIT')) {\n    // Page budget exhausted: surface partial-data health, do not publish truncated set\n    reportHealth('eccc', { ok: false, reason: 'page-limit', detail: err.message });\n  } else throw err;\n}","preventionTips":["Monitor numberMatched on the ECCC collection and alert when it approaches ECCC_MAX_PAGES * ECCC_PAGE_SIZE (2000).","Remember one failed status is tolerated (partial result) — only rely on the hard throw when both statuses fail.","In tests, make fetch stubs honor offset/limit so paging always terminates.","After ECCC API version changes, re-verify limit/offset behavior before deploying."],"tags":["pagination","api","limit-exceeded","weather"],"backgroundTag":"value-out-of-range","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}