{"record":{"id":"29a7df874f6e834f","repo":"koala73/worldmonitor","slug":"eccc-page-progress","errorCode":"ECCC_PAGE_PROGRESS","errorMessage":"ECCC_PAGE_PROGRESS","messagePattern":"ECCC_PAGE_PROGRESS","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/_weather-alert-select.mjs","lineNumber":781,"sourceCode":"        if (byteBudget.remaining <= 0) throw new Error('ECCC_AGGREGATE_TOO_LARGE');\n        const url = new URL(ECCC_ALERTS_URLS[index]);\n        url.searchParams.set('offset', String(statusFeatures.length));\n        pages += 1;\n        const data = await fetchApprovedWeatherJson(url.toString(), {\n          allowedHosts: [ECCC_HOST], maxBytes, fetchFn, userAgent, byteBudget,\n        });\n        const page = requireAlertFeatures(data);\n        if (data.type !== 'FeatureCollection'\n          || !Number.isSafeInteger(data.numberMatched) || data.numberMatched < 0\n          || !Number.isSafeInteger(data.numberReturned) || data.numberReturned !== page.length\n          || page.length > ECCC_PAGE_SIZE) {\n          throw new Error('ECCC_MALFORMED_PAGE');\n        }\n        if (matched !== undefined && data.numberMatched !== matched) throw new Error('ECCC_COUNT_DRIFT');\n        matched = data.numberMatched;\n        if (statusFeatures.length + page.length > matched\n          || (page.length === 0 && statusFeatures.length < matched)) {\n          throw new Error('ECCC_PAGE_PROGRESS');\n        }\n        for (const feature of page) {\n          if (typeof feature?.id !== 'string' || !feature.id.trim()) throw new Error('ECCC_INVALID_ID');\n          if (seenIds.has(feature.id)) throw new Error('ECCC_DUPLICATE_ID');\n          seenIds.add(feature.id);\n        }\n        statusFeatures.push(...page);\n      } while (statusFeatures.length < matched);\n      features.push(...statusFeatures);\n    } catch (err) {\n      failures.push(err);\n      failedStatuses.push(status);\n    }\n  }\n  if (failures.length === ECCC_LIVE_STATUSES.length) {\n    const detail = failures.map((err) => err?.message || String(err)).join('; ');\n    throw new Error(`ECCC issued and continued fetches both failed: ${detail}`);\n  }","sourceCodeStart":763,"sourceCodeEnd":799,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/_weather-alert-select.mjs#L763-L799","documentation":"The paging loop requires monotonic, truthful progress: accumulated features plus the current page must not exceed numberMatched, and a zero-length page is only allowed once all matched features have been collected (statusFeatures.length === matched). A page that overshoots the declared total, or an empty page returned while features are still outstanding, indicates broken pagination semantics, so Error('ECCC_PAGE_PROGRESS') is thrown rather than risking a truncated or looped result.","triggerScenarios":"In scripts/_weather-alert-select.mjs:781: (a) statusFeatures.length + page.length > matched — the server returned more features than numberMatched promised (e.g. duplicates pushed by a concurrent mutation or an offset that restarted), or (b) page.length === 0 while statusFeatures.length < matched — the server returned an empty page before the collection was fully drained.","commonSituations":"An upstream OGC API regression where offset paging returns empty pages mid-collection; alerts expiring between pages so the offset window shifts and the final page over-collects relative to the (stale) numberMatched; a proxy caching an older page and replaying it, breaking forward progress; a test stub with incorrect offset handling.","solutions":["Query the collection manually with successive offsets to reproduce which page breaks: ...?limit=250&offset=N; confirm whether the upstream server skips or repeats items.","Retry the fetch — if alerts changed mid-pagination causing the overshoot, a fresh run with a fresh numberMatched usually succeeds.","If the ECCC API regressed on offset paging, consider switching the pagination key (sortby=feature_id is already set) or paging on the returned feature ids instead of raw offsets.","Fix test stubs to implement offset correctly: return exactly `limit` features per page from the sorted dataset and an empty page only at/after the end."],"exampleFix":"// before: stub ignores offset and always returns page 1\nconst page = allFeatures.slice(0, limit);\n// after: honor offset so progress is monotonic\nconst page = allFeatures.slice(offset, offset + limit);","handlingStrategy":"retry","validationCode":"function pageIsProgressive(accumulated, page, matched) {\n  if (accumulated + page.length > matched) return false;\n  if (page.length === 0 && accumulated < matched) return false;\n  return true;\n}","typeGuard":null,"tryCatchPattern":"async function fetchEcccSafe(opts) {\n  try {\n    return await fetchEcccAlertFeatures(opts);\n  } catch (err) {\n    if (String(err.message).includes('ECCC_PAGE_PROGRESS')) {\n      // Upstream pagination skipped/overshot; one retry usually lands on a consistent snapshot\n      return fetchEcccAlertFeatures(opts);\n    }\n    throw err;\n  }\n}","preventionTips":["Keep sortby=feature_id in ECCC_ALERTS_URLS requests — stable ordering is what makes offset paging sound.","Make test stubs implement offset slicing exactly; empty-before-end and overshoot are the classic stub bugs.","Verify result.partial / failedStatuses before publishing; one failed status still returns a usable object.","If the upstream API regresses on offset paging, switch pagination strategy rather than disabling the progress check."],"tags":["pagination","consistency","ogc-api","upstream"],"backgroundTag":"unexpected-response-shape","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}