{"record":{"id":"99b1dc4424db578b","repo":"koala73/worldmonitor","slug":"published-sitemap-tree-exceeds-50-documents","errorCode":null,"errorMessage":"published sitemap tree exceeds 50 documents","messagePattern":"published sitemap tree exceeds 50 documents","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/seo-indexnow-submit.mjs","lineNumber":174,"sourceCode":"  // AI-crawler stub surface middleware.ts serves, so submit both.\n  ...INDEXNOW_VARIANT_HOSTS.map((host) => batch(host, urlsForHost(host, [`https://${host}/`]))),\n]);\n\nexport async function getPublishedBatches({ fetchImpl = globalThis.fetch } = {}) {\n  const origin = SITE_ORIGIN;\n  const pending = [`${origin}/sitemap.xml`];\n  const seen = new Set();\n  const pages = new Set();\n  const hosts = new Set(INDEXNOW_BATCHES.map(config => config.host));\n  while (pending.length > 0) {\n    const location = pending.shift();\n    const sitemapUrl = new URL(location);\n    if (sitemapUrl.origin !== origin || sitemapUrl.username || sitemapUrl.password || sitemapUrl.search || sitemapUrl.hash || !sitemapUrl.pathname.endsWith('.xml')) {\n      throw new Error(`invalid sitemap location: ${location}`);\n    }\n    if (seen.has(location)) continue;\n    seen.add(location);\n    if (seen.size > 50) throw new Error('published sitemap tree exceeds 50 documents');\n    const response = await fetchImpl(location, {\n      method: 'GET',\n      redirect: 'manual',\n      signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),\n      headers: { Accept: 'application/xml', 'User-Agent': USER_AGENT },\n    });\n    if (response.status !== 200) throw new Error(`${location} returned ${response.status}, expected direct 200`);\n    const source = (await response.text()).trim();\n    const root = /^(?:<\\?xml[^?]*\\?>\\s*)?<(sitemapindex|urlset)\\b[^>]*>([\\s\\S]*)<\\/\\1>\\s*$/.exec(source);\n    if (!root) throw new Error(`invalid sitemap document: ${location}`);\n    const tag = root[1] === 'sitemapindex' ? 'sitemap' : 'url';\n    const entryPattern = new RegExp(`<!--[\\\\s\\\\S]*?-->|<${tag}>[\\\\s\\\\S]*?<\\\\/${tag}>`, 'g');\n    const entries = [...root[2].matchAll(entryPattern)].filter(([entry]) => !entry.startsWith('<!--'));\n    if (root[2].replace(entryPattern, '').trim()) throw new Error(`invalid sitemap entries: ${location}`);\n    const urls = entries.map(([entry]) => {\n      const locations = [...entry.matchAll(/<loc>\\s*([^<]+?)\\s*<\\/loc>/g)];\n      if (locations.length !== 1) throw new Error(`expected one location per sitemap entry: ${location}`);\n      return decodeXml(locations[0][1].trim());","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/seo-indexnow-submit.mjs#L156-L192","documentation":"As getPublishedBatches walks the sitemap tree it tracks visited locations in a seen set and enforces a hard cap of 50 documents. If discovering more sitemaps would exceed that cap the script throws 'published sitemap tree exceeds 50 documents', a safety limit preventing runaway crawls of accidentally huge or hostile sitemap indexes.","triggerScenarios":"The published site exposes more than 50 distinct sitemap documents reachable via nested sitemapindex files — e.g. an automated sitemap generator shard-per-day, duplicated index chains, or a loop where different URLs point to overlapping sitemaps.","commonSituations":"Sitemap generator creating per-day/per-tag sitemaps that have accumulated over months; nested sitemapindexes referencing each other; staging data copied to production inflating sitemap count; pagination generating thousands of shards.","solutions":["Audit and prune the sitemap tree: consolidate shards into fewer sitemaps or remove stale ones","Add a sitemap index flattening step / raise the cap in the script only if >50 documents is genuinely correct","Ensure the sitemap generator deduplicates and does not emit self-referential index loops","Clean up old dated sitemaps and regenerate the index"],"exampleFix":"// before\nif (seen.size > 50) throw new Error('published sitemap tree exceeds 50 documents');\n// after\nconst MAX_SITEMAPS = Number(process.env.MAX_SITEMAP_DOCS ?? 50);\nif (seen.size > MAX_SITEMAPS) throw new Error(`published sitemap tree exceeds ${MAX_SITEMAPS} documents`);","handlingStrategy":"fallback","validationCode":"const count = await countPublishedSitemaps(indexUrl);\nif (count > 50) console.warn(`sitemap tree has ${count} docs; cap is 50`);","typeGuard":null,"tryCatchPattern":"try {\n  await getPublishedBatches();\n} catch (err) {\n  if (err.message === 'published sitemap tree exceeds 50 documents') {\n    await submitKnownBatchesOnly();\n  } else throw err;\n}","preventionTips":["Consolidate sitemap shards regularly instead of accumulating per-day files","Monitor sitemap document count and alert before hitting 50","Deduplicate nested sitemapindex references"],"tags":["seo","sitemap","limit-exceeded"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}