koala73/worldmonitor · error

Pulse freeze selected

Error message

Pulse freeze selected ${snapshot.headlines.length} headlines carrying a repeated article: ${duplicateHeadlineUrls.join(', ')}

What it means

The crawlable live-pulse freeze script enforces that the selected homepage headlines contain no duplicated article URLs: duplicates would place one story in two of the four homepage rows, which the script treats as a dedupe defect, not an upstream condition. If duplicateArticleUrls finds any repeated URL among the selected headlines, it throws listing the count and the duplicated URLs. Unlike coverage gates, this cannot be caused by a news outage, so it throws rather than recording a partial freeze.

Solutions

1. Inspect the listed duplicate URLs and fix the upstream dedupe so each article URL appears once.
2. Normalize URLs (strip tracking params, trailing slashes, canonicalize) before deduping.
3. Dedupe across the full selected set rather than per-row/per-category.
4. Re-run the freeze after the dedupe fix; do not ship a snapshot with duplicated stories.

Example fix

// before
const seen = new Set(rows.map((r) => r.url)); // per category
// after
const seen = new Set(allRows.map((r) => normalizeUrl(r.url))); // global, normalized

const urls = headlines.map((h) => normalizeUrl(h.url));
if (new Set(urls).size !== urls.length) throw new Error('duplicate article URLs in selection');

try { await freezePulse(); } catch (e) { if (e.message.includes('repeated article')) { console.error(e.message); process.exitCode = 1; } else throw e; }

When it happens

Trigger: Running the freeze script when the dedupe step upstream fails to remove a story that appears in multiple feed rows, e.g. the same article URL selected under two categories or from two feeds.

Common situations: A publisher syndicating one article across multiple feeds/categories with the same canonical URL; URL normalization drift (trailing slash, query params) defeating set-based dedupe; upstream feed changes introducing cross-category overlap.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• classify-eval: could not find
• crawlable corpus captured dated country developments for
• Ok physical divergence reading has an invalid index

AI-assisted analysis of koala73/worldmonitor@e586b8b4b8 (2026-09-22). Data as JSON: /api/errors/fbc81434a93e64cc. Report an issue: GitHub.

Appendix: source

Thrown at scripts/freeze-crawlable-live-pulse.mjs:1382

    const minBriefs = minimumBriefCaptures(snapshot.coverage.briefMatchedCount);
    if (checkedBriefs < minBriefs) {
      throw new Error(
        `Pulse freeze captured or withheld unsupported briefs for only ${checkedBriefs} of ${snapshot.coverage.briefMatchedCount} headline-matched countries; `
        + `expected at least ${minBriefs}`
        + firstCaptureCause(developmentsErrors),
      );
    }
  }

  // Strip invariant (#8339): the four published headlines must be four distinct
  // articles. selectFrozenHeadlines already dedupes by normalized URL, so a
  // duplicate here is a defect in that dedupe rather than an upstream
  // condition, and it would put one story in two of the homepage's four rows.
  // Unlike the coverage gates above this cannot be caused by a news outage, so
  // it throws instead of recording a partial.
  const duplicateHeadlineUrls = duplicateArticleUrls(snapshot.headlines, (row) => row?.url);
  if (duplicateHeadlineUrls.length > 0) {
    throw new Error(
      `Pulse freeze selected ${snapshot.headlines.length} headlines carrying a repeated article: `
      + `${duplicateHeadlineUrls.join(', ')}`,
    );
  }

  const basename = OUTPUT_BASENAME || `crawlable-live-pulse-${capturedAt}.json`;
  const outPath = path.join(rootDir, 'docs', 'snapshots', basename);
  await fs.writeFile(outPath, `${JSON.stringify(snapshot, null, 2)}\n`, 'utf8');
  return { outPath, snapshot };
}

const isMain = process.argv[1] && path.resolve(process.argv[1]) === __filename;
if (isMain) {
  freezeCrawlableLivePulse()
    .then(({ outPath, snapshot }) => {
      console.log(`[freeze-crawlable-live-pulse] wrote ${outPath}`);
      console.log(
        `[freeze-crawlable-live-pulse] countries=${snapshot.coverage.countryCount} `

View on GitHub (pinned to e586b8b4b8)