{"record":{"id":"84fedf0a9be4d8e1","repo":"apify/crawlee","slug":"unsupported-sitemap-content-type-contenttype","errorCode":null,"errorMessage":"Unsupported sitemap content type (contentType = ${contentType}, url = ${url?.toString()})","messagePattern":"Unsupported sitemap content type \\(contentType = (.+?), url = (.+?)\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/utils/src/internals/sitemap.ts","lineNumber":270,"sourceCode":"\n    const createParser = async (contentType = '', url?: URL): Promise<Duplex> => {\n        let mimeType: MIMEType | null;\n\n        try {\n            mimeType = new MIMEType(contentType);\n        } catch {\n            mimeType = null;\n        }\n\n        if (mimeType?.isXML() || url?.pathname.endsWith('.xml')) {\n            return SitemapXmlParser.create();\n        }\n\n        if (mimeType?.essence === 'text/plain' || url?.pathname.endsWith('.txt')) {\n            return new SitemapTxtParser();\n        }\n\n        throw new Error(`Unsupported sitemap content type (contentType = ${contentType}, url = ${url?.toString()})`);\n    };\n\n    while (sources.length > 0) {\n        const source = sources.shift()!;\n\n        if ((source?.depth ?? 0) > maxDepth) {\n            continue;\n        }\n\n        let items: AsyncIterable<SitemapItem> | null = null;\n\n        // Parent URL, parsed once and reused as the origin for the strategy checks below.\n        let sitemapUrl: URL | undefined;\n\n        if (source.type === 'url') {\n            sitemapUrl = new URL(source.url);\n            visitedSitemapUrls.add(sitemapUrl.toString());\n            let retriesLeft = sitemapRetries + 1;","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/utils/src/internals/sitemap.ts#L252-L288","documentation":"createParser dispatches a sitemap parser based on Content-Type or URL extension (XML, gzip, text/plain). If the content type's MIME essence and URL extension match none of the supported kinds, it throws this error rather than parsing garbage. It means the response is not a recognizable sitemap format.","triggerScenarios":"Calling parseSitemap()/SitemapUrlListLoader on a URL whose response Content-Type is something like 'application/json' or 'text/html' and whose path does not end in .xml, .gz, or .txt — e.g. an HTML error page or a REST endpoint instead of a sitemap.","commonSituations":"robots.txt pointing to a sitemap index that returns an HTML 404/soft-404 page; CDN serving HTML challenge pages; sitemap behind an endpoint that ignores Accept headers; typo'd sitemap URL; server sending an unlisted Content-Type like 'application/octet-stream'.","solutions":["Verify the sitemap URL actually returns sitemap XML/TXT (curl -I to check Content-Type)","Check the response isn't a soft-404 or bot-challenge HTML page","Serve sitemaps with correct Content-Type (application/xml, text/plain, application/x-gzip)","If content is correct, fix server MIME config so the essence matches a supported type","Skip/queue the source and continue with remaining sitemap sources in a try-catch"],"exampleFix":"// before\nconst urls = await parseSitemap([sitemapUrl]);\n\n// after\ntry {\n  const urls = await parseSitemap([sitemapUrl]);\n} catch (err) {\n  if (String(err).includes('Unsupported sitemap content type')) {\n    log.warning(`skipping ${sitemapUrl}: not a sitemap`);\n  } else throw err;\n}","handlingStrategy":"try-catch","validationCode":"const res = await fetch(sitemapUrl, { method: 'HEAD' });\nconst ct = res.headers.get('content-type') ?? '';\nconst ok = /xml|gzip|text\\/plain|octet-stream/.test(ct) || sitemapUrl.endsWith('.xml') || sitemapUrl.endsWith('.txt');\nif (!ok) console.warn(`${sitemapUrl} is probably not a sitemap (${ct})`);","typeGuard":"function looksLikeSitemapUrl(u: string): boolean {\n  return /\\.(xml|gz|txt)(\\?|$)/.test(u);\n}","tryCatchPattern":"try {\n  urls = await parseSitemap([sitemapUrl]);\n} catch (err) {\n  if (String(err).includes('Unsupported sitemap content type')) urls = [];\n  else throw err;\n}","preventionTips":["HEAD-check Content-Type before parsing a sitemap URL","Skip sources listed in robots.txt that return HTML (soft-404/challenge pages)","Serve sitemaps with correct MIME types if you control the server","Handle sitemap index files rather than assuming every listed URL is a sitemap"],"tags":["sitemap","content-type","parsing"],"backgroundTag":"unsupported-content-type","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}