koala73/worldmonitor · error · Error

published sitemap tree exceeds 50 documents

Error message

published sitemap tree exceeds 50 documents

What it means

As getPublishedBatches walks the sitemap tree it tracks visited locations in a seen set and enforces a hard cap of 50 documents. If discovering more sitemaps would exceed that cap the script throws 'published sitemap tree exceeds 50 documents', a safety limit preventing runaway crawls of accidentally huge or hostile sitemap indexes.

Solutions

  1. Audit and prune the sitemap tree: consolidate shards into fewer sitemaps or remove stale ones
  2. Add a sitemap index flattening step / raise the cap in the script only if >50 documents is genuinely correct
  3. Ensure the sitemap generator deduplicates and does not emit self-referential index loops
  4. Clean up old dated sitemaps and regenerate the index

Example fix

// before
if (seen.size > 50) throw new Error('published sitemap tree exceeds 50 documents');
// after
const MAX_SITEMAPS = Number(process.env.MAX_SITEMAP_DOCS ?? 50);
if (seen.size > MAX_SITEMAPS) throw new Error(`published sitemap tree exceeds ${MAX_SITEMAPS} documents`);
Defensive patterns

Strategy: fallback

Validate before calling

const count = await countPublishedSitemaps(indexUrl);
if (count > 50) console.warn(`sitemap tree has ${count} docs; cap is 50`);

Try / catch

try {
  await getPublishedBatches();
} catch (err) {
  if (err.message === 'published sitemap tree exceeds 50 documents') {
    await submitKnownBatchesOnly();
  } else throw err;
}

Prevention

When it happens

Trigger: The published site exposes more than 50 distinct sitemap documents reachable via nested sitemapindex files — e.g. an automated sitemap generator shard-per-day, duplicated index chains, or a loop where different URLs point to overlapping sitemaps.

Common situations: Sitemap generator creating per-day/per-tag sitemaps that have accumulated over months; nested sitemapindexes referencing each other; staging data copied to production inflating sitemap count; pagination generating thousands of shards.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of koala73/worldmonitor@7d06c8633d (2026-09-15). Data as JSON: /api/errors/99b1dc4424db578b. Report an issue: GitHub.

Appendix: source

Thrown at scripts/seo-indexnow-submit.mjs:174

  // AI-crawler stub surface middleware.ts serves, so submit both.
  ...INDEXNOW_VARIANT_HOSTS.map((host) => batch(host, urlsForHost(host, [`https://${host}/`]))),
]);

export async function getPublishedBatches({ fetchImpl = globalThis.fetch } = {}) {
  const origin = SITE_ORIGIN;
  const pending = [`${origin}/sitemap.xml`];
  const seen = new Set();
  const pages = new Set();
  const hosts = new Set(INDEXNOW_BATCHES.map(config => config.host));
  while (pending.length > 0) {
    const location = pending.shift();
    const sitemapUrl = new URL(location);
    if (sitemapUrl.origin !== origin || sitemapUrl.username || sitemapUrl.password || sitemapUrl.search || sitemapUrl.hash || !sitemapUrl.pathname.endsWith('.xml')) {
      throw new Error(`invalid sitemap location: ${location}`);
    }
    if (seen.has(location)) continue;
    seen.add(location);
    if (seen.size > 50) throw new Error('published sitemap tree exceeds 50 documents');
    const response = await fetchImpl(location, {
      method: 'GET',
      redirect: 'manual',
      signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
      headers: { Accept: 'application/xml', 'User-Agent': USER_AGENT },
    });
    if (response.status !== 200) throw new Error(`${location} returned ${response.status}, expected direct 200`);
    const source = (await response.text()).trim();
    const root = /^(?:<\?xml[^?]*\?>\s*)?<(sitemapindex|urlset)\b[^>]*>([\s\S]*)<\/\1>\s*$/.exec(source);
    if (!root) throw new Error(`invalid sitemap document: ${location}`);
    const tag = root[1] === 'sitemapindex' ? 'sitemap' : 'url';
    const entryPattern = new RegExp(`<!--[\\s\\S]*?-->|<${tag}>[\\s\\S]*?<\\/${tag}>`, 'g');
    const entries = [...root[2].matchAll(entryPattern)].filter(([entry]) => !entry.startsWith('<!--'));
    if (root[2].replace(entryPattern, '').trim()) throw new Error(`invalid sitemap entries: ${location}`);
    const urls = entries.map(([entry]) => {
      const locations = [...entry.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)];
      if (locations.length !== 1) throw new Error(`expected one location per sitemap entry: ${location}`);
      return decodeXml(locations[0][1].trim());

View on GitHub (pinned to 7d06c8633d)