koala73/worldmonitor · error · Error

invalid sitemap document: ${location}

Error message

invalid sitemap document: ${location}

What it means

After a 200 fetch the script validates the response body is a well-formed sitemap: an optional XML declaration followed by exactly one <sitemapindex> or <urlset> root element. If the body does not match this structure the script throws 'invalid sitemap document: <location>' instead of parsing arbitrary HTML or error pages.

Solutions

  1. Fetch the URL manually and confirm the body is a valid <sitemapindex> or <urlset> document
  2. Fix whatever serves HTML/error pages with 200 at that URL (WAF interstitial, SPA fallback for .xml routes)
  3. Verify the sitemap generator emits a single well-formed root element with XML declaration only
  4. Check Content-Encoding handling so the script receives decompressed XML

Example fix

// before (soft-404 with 200)
<html><body>Not Found</body></html>
// after
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">...</urlset>
Defensive patterns

Strategy: validation

Validate before calling

const body = (await (await fetch(url)).text()).trim();
if (!/^<\?xml|<(sitemapindex|urlset)\b/.test(body)) console.warn(`${url} is not a sitemap document`);

Type guard

const isSitemapDoc = (source) => /^(?:<\?xml[^?]*\?>\s*)?<(sitemapindex|urlset)\b[^>]*>[\s\S]*<\/(sitemapindex|urlset)>\s*$/.test(source.trim());

Try / catch

try {
  await getPublishedBatches();
} catch (err) {
  if (err.message.startsWith('invalid sitemap document:')) {
    const loc = err.message.split(': ').slice(1).join(': ');
    logger.error({ loc }, 'non-XML 200 body; check WAF soft-404 or generator output');
  } else throw err;
}

Prevention

When it happens

Trigger: The 200 response body is HTML (a soft-404/captcha/login page served with 200), an XML parse error page, an empty body, gzip bytes not decompressed, extra content outside the root element, or misspelled root tags.

Common situations: WAF serving an interstitial with 200; server returning HTML error page with 200 status; compression middleware breaking XML; a generator emitting fragment/invalid XML; content-type confusion returning JSON instead of XML.

Related errors

AI-assisted analysis of koala73/worldmonitor@7d06c8633d (2026-09-15). Data as JSON: /api/errors/62e3d28332e2a033. Report an issue: GitHub.

Appendix: source

Thrown at scripts/seo-indexnow-submit.mjs:184

  while (pending.length > 0) {
    const location = pending.shift();
    const sitemapUrl = new URL(location);
    if (sitemapUrl.origin !== origin || sitemapUrl.username || sitemapUrl.password || sitemapUrl.search || sitemapUrl.hash || !sitemapUrl.pathname.endsWith('.xml')) {
      throw new Error(`invalid sitemap location: ${location}`);
    }
    if (seen.has(location)) continue;
    seen.add(location);
    if (seen.size > 50) throw new Error('published sitemap tree exceeds 50 documents');
    const response = await fetchImpl(location, {
      method: 'GET',
      redirect: 'manual',
      signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
      headers: { Accept: 'application/xml', 'User-Agent': USER_AGENT },
    });
    if (response.status !== 200) throw new Error(`${location} returned ${response.status}, expected direct 200`);
    const source = (await response.text()).trim();
    const root = /^(?:<\?xml[^?]*\?>\s*)?<(sitemapindex|urlset)\b[^>]*>([\s\S]*)<\/\1>\s*$/.exec(source);
    if (!root) throw new Error(`invalid sitemap document: ${location}`);
    const tag = root[1] === 'sitemapindex' ? 'sitemap' : 'url';
    const entryPattern = new RegExp(`<!--[\\s\\S]*?-->|<${tag}>[\\s\\S]*?<\\/${tag}>`, 'g');
    const entries = [...root[2].matchAll(entryPattern)].filter(([entry]) => !entry.startsWith('<!--'));
    if (root[2].replace(entryPattern, '').trim()) throw new Error(`invalid sitemap entries: ${location}`);
    const urls = entries.map(([entry]) => {
      const locations = [...entry.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)];
      if (locations.length !== 1) throw new Error(`expected one location per sitemap entry: ${location}`);
      return decodeXml(locations[0][1].trim());
    });
    if (urls.length === 0) throw new Error(`empty sitemap: ${location}`);
    if (location === `${origin}/sitemap.xml`
      && (root[1] !== 'sitemapindex' || urls.length !== SITEMAP_INDEX_MEMBERS.length
        || new Set(urls).size !== urls.length || urls.some(url => !SITEMAP_INDEX_MEMBERS.includes(url)))) {
      throw new Error('published root sitemap members do not match the declared inventory');
    }
    if (root[1] === 'sitemapindex') {
      pending.push(...urls);
    } else {

View on GitHub (pinned to 7d06c8633d)