{"record":{"id":"0931403ee6454fdd","repo":"koala73/worldmonitor","slug":"location-returned-response-status-expected-direct-200","errorCode":null,"errorMessage":"${location} returned ${response.status}, expected direct 200","messagePattern":"(.+?) returned (.+?), expected direct 200","errorType":"http","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/seo-indexnow-submit.mjs","lineNumber":181,"sourceCode":"  const seen = new Set();\n  const pages = new Set();\n  const hosts = new Set(INDEXNOW_BATCHES.map(config => config.host));\n  while (pending.length > 0) {\n    const location = pending.shift();\n    const sitemapUrl = new URL(location);\n    if (sitemapUrl.origin !== origin || sitemapUrl.username || sitemapUrl.password || sitemapUrl.search || sitemapUrl.hash || !sitemapUrl.pathname.endsWith('.xml')) {\n      throw new Error(`invalid sitemap location: ${location}`);\n    }\n    if (seen.has(location)) continue;\n    seen.add(location);\n    if (seen.size > 50) throw new Error('published sitemap tree exceeds 50 documents');\n    const response = await fetchImpl(location, {\n      method: 'GET',\n      redirect: 'manual',\n      signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),\n      headers: { Accept: 'application/xml', 'User-Agent': USER_AGENT },\n    });\n    if (response.status !== 200) throw new Error(`${location} returned ${response.status}, expected direct 200`);\n    const source = (await response.text()).trim();\n    const root = /^(?:<\\?xml[^?]*\\?>\\s*)?<(sitemapindex|urlset)\\b[^>]*>([\\s\\S]*)<\\/\\1>\\s*$/.exec(source);\n    if (!root) throw new Error(`invalid sitemap document: ${location}`);\n    const tag = root[1] === 'sitemapindex' ? 'sitemap' : 'url';\n    const entryPattern = new RegExp(`<!--[\\\\s\\\\S]*?-->|<${tag}>[\\\\s\\\\S]*?<\\\\/${tag}>`, 'g');\n    const entries = [...root[2].matchAll(entryPattern)].filter(([entry]) => !entry.startsWith('<!--'));\n    if (root[2].replace(entryPattern, '').trim()) throw new Error(`invalid sitemap entries: ${location}`);\n    const urls = entries.map(([entry]) => {\n      const locations = [...entry.matchAll(/<loc>\\s*([^<]+?)\\s*<\\/loc>/g)];\n      if (locations.length !== 1) throw new Error(`expected one location per sitemap entry: ${location}`);\n      return decodeXml(locations[0][1].trim());\n    });\n    if (urls.length === 0) throw new Error(`empty sitemap: ${location}`);\n    if (location === `${origin}/sitemap.xml`\n      && (root[1] !== 'sitemapindex' || urls.length !== SITEMAP_INDEX_MEMBERS.length\n        || new Set(urls).size !== urls.length || urls.some(url => !SITEMAP_INDEX_MEMBERS.includes(url)))) {\n      throw new Error('published root sitemap members do not match the declared inventory');\n    }","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/seo-indexnow-submit.mjs#L163-L199","documentation":"getPublishedBatches fetches each sitemap location with redirect: 'manual' and requires a direct HTTP 200. Any other status (301/302 redirect, 404, 403, 5xx) throws '<location> returned <status>, expected direct 200', because sitemap URLs must be directly reachable at their canonical address for IndexNow submission.","triggerScenarios":"The sitemap URL responds with a redirect (moved to www/HTTPS, trailing-slash normalization), a 404 from a stale <loc> entry, 403 from WAF/bot protection blocking the script's User-Agent, or a 5xx during deploys.","commonSituations":"Sitemap <loc> URLs published before a domain/migration change now redirecting; CDN or bot protection rejecting the script's User-Agent; deploy window where sitemaps briefly 404; HTTP→HTTPS or apex→www redirects.","solutions":["Fix the published sitemap <loc> URLs to point at the final, directly-200 canonical addresses","Whitelist the script's User-Agent in the WAF/CDN if 403 is returned","Re-run the script outside deploy windows or after cache warm-up if 5xx/404 was transient","Verify no redirect rules (http→https, apex→www, trailing slash) rewrite sitemap URLs and publish the resolved form"],"exampleFix":"// before\n<loc>http://example.com/sitemaps/pages.xml</loc>  <!-- 301 → https://www -->\n// after\n<loc>https://www.example.com/sitemaps/pages.xml</loc>  <!-- direct 200 -->","handlingStrategy":"retry","validationCode":"const res = await fetch(url, { redirect: 'manual' });\nif (res.status !== 200) console.warn(`${url} is not directly reachable: ${res.status}`);","typeGuard":null,"tryCatchPattern":"try {\n  await getPublishedBatches();\n} catch (err) {\n  if (/expected direct 200$/.test(err.message)) {\n    const status = Number(err.message.match(/returned (\\d+)/)?.[1]);\n    if ([429, 500, 502, 503, 504].includes(status)) await retryWithBackoff();\n    else logger.error({ url: err.message.split(' returned')[0], status }, 'sitemap permanently unreachable');\n  } else throw err;\n}","preventionTips":["Publish final canonical sitemap URLs (no redirects)","Allowlist the crawler User-Agent in WAF/CDN rules","Avoid running submissions during deploys when sitemaps may briefly 404"],"tags":["seo","sitemap","http"],"backgroundTag":"http-non-200-response","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}