koala73/worldmonitor · error · Error

${location} returned ${response.status}, expected direct 200

Error message

${location} returned ${response.status}, expected direct 200

What it means

getPublishedBatches fetches each sitemap location with redirect: 'manual' and requires a direct HTTP 200. Any other status (301/302 redirect, 404, 403, 5xx) throws '<location> returned <status>, expected direct 200', because sitemap URLs must be directly reachable at their canonical address for IndexNow submission.

Solutions

  1. Fix the published sitemap <loc> URLs to point at the final, directly-200 canonical addresses
  2. Whitelist the script's User-Agent in the WAF/CDN if 403 is returned
  3. Re-run the script outside deploy windows or after cache warm-up if 5xx/404 was transient
  4. Verify no redirect rules (http→https, apex→www, trailing slash) rewrite sitemap URLs and publish the resolved form

Example fix

// before
<loc>http://example.com/sitemaps/pages.xml</loc>  <!-- 301 → https://www -->
// after
<loc>https://www.example.com/sitemaps/pages.xml</loc>  <!-- direct 200 -->
Defensive patterns

Strategy: retry

Validate before calling

const res = await fetch(url, { redirect: 'manual' });
if (res.status !== 200) console.warn(`${url} is not directly reachable: ${res.status}`);

Try / catch

try {
  await getPublishedBatches();
} catch (err) {
  if (/expected direct 200$/.test(err.message)) {
    const status = Number(err.message.match(/returned (\d+)/)?.[1]);
    if ([429, 500, 502, 503, 504].includes(status)) await retryWithBackoff();
    else logger.error({ url: err.message.split(' returned')[0], status }, 'sitemap permanently unreachable');
  } else throw err;
}

Prevention

When it happens

Trigger: The sitemap URL responds with a redirect (moved to www/HTTPS, trailing-slash normalization), a 404 from a stale <loc> entry, 403 from WAF/bot protection blocking the script's User-Agent, or a 5xx during deploys.

Common situations: Sitemap <loc> URLs published before a domain/migration change now redirecting; CDN or bot protection rejecting the script's User-Agent; deploy window where sitemaps briefly 404; HTTP→HTTPS or apex→www redirects.

Related errors

AI-assisted analysis of koala73/worldmonitor@7d06c8633d (2026-09-15). Data as JSON: /api/errors/0931403ee6454fdd. Report an issue: GitHub.

Appendix: source

Thrown at scripts/seo-indexnow-submit.mjs:181

  const seen = new Set();
  const pages = new Set();
  const hosts = new Set(INDEXNOW_BATCHES.map(config => config.host));
  while (pending.length > 0) {
    const location = pending.shift();
    const sitemapUrl = new URL(location);
    if (sitemapUrl.origin !== origin || sitemapUrl.username || sitemapUrl.password || sitemapUrl.search || sitemapUrl.hash || !sitemapUrl.pathname.endsWith('.xml')) {
      throw new Error(`invalid sitemap location: ${location}`);
    }
    if (seen.has(location)) continue;
    seen.add(location);
    if (seen.size > 50) throw new Error('published sitemap tree exceeds 50 documents');
    const response = await fetchImpl(location, {
      method: 'GET',
      redirect: 'manual',
      signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
      headers: { Accept: 'application/xml', 'User-Agent': USER_AGENT },
    });
    if (response.status !== 200) throw new Error(`${location} returned ${response.status}, expected direct 200`);
    const source = (await response.text()).trim();
    const root = /^(?:<\?xml[^?]*\?>\s*)?<(sitemapindex|urlset)\b[^>]*>([\s\S]*)<\/\1>\s*$/.exec(source);
    if (!root) throw new Error(`invalid sitemap document: ${location}`);
    const tag = root[1] === 'sitemapindex' ? 'sitemap' : 'url';
    const entryPattern = new RegExp(`<!--[\\s\\S]*?-->|<${tag}>[\\s\\S]*?<\\/${tag}>`, 'g');
    const entries = [...root[2].matchAll(entryPattern)].filter(([entry]) => !entry.startsWith('<!--'));
    if (root[2].replace(entryPattern, '').trim()) throw new Error(`invalid sitemap entries: ${location}`);
    const urls = entries.map(([entry]) => {
      const locations = [...entry.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)];
      if (locations.length !== 1) throw new Error(`expected one location per sitemap entry: ${location}`);
      return decodeXml(locations[0][1].trim());
    });
    if (urls.length === 0) throw new Error(`empty sitemap: ${location}`);
    if (location === `${origin}/sitemap.xml`
      && (root[1] !== 'sitemapindex' || urls.length !== SITEMAP_INDEX_MEMBERS.length
        || new Set(urls).size !== urls.length || urls.some(url => !SITEMAP_INDEX_MEMBERS.includes(url)))) {
      throw new Error('published root sitemap members do not match the declared inventory');
    }

View on GitHub (pinned to 7d06c8633d)