apify/crawlee · error

Unsupported sitemap content type (contentType = ${contentTyp

Error message

Unsupported sitemap content type (contentType = ${contentType}, url = ${url?.toString()})

What it means

createParser dispatches a sitemap parser based on Content-Type or URL extension (XML, gzip, text/plain). If the content type's MIME essence and URL extension match none of the supported kinds, it throws this error rather than parsing garbage. It means the response is not a recognizable sitemap format.

Source

Thrown at packages/utils/src/internals/sitemap.ts:270

    const createParser = async (contentType = '', url?: URL): Promise<Duplex> => {
        let mimeType: MIMEType | null;

        try {
            mimeType = new MIMEType(contentType);
        } catch {
            mimeType = null;
        }

        if (mimeType?.isXML() || url?.pathname.endsWith('.xml')) {
            return SitemapXmlParser.create();
        }

        if (mimeType?.essence === 'text/plain' || url?.pathname.endsWith('.txt')) {
            return new SitemapTxtParser();
        }

        throw new Error(`Unsupported sitemap content type (contentType = ${contentType}, url = ${url?.toString()})`);
    };

    while (sources.length > 0) {
        const source = sources.shift()!;

        if ((source?.depth ?? 0) > maxDepth) {
            continue;
        }

        let items: AsyncIterable<SitemapItem> | null = null;

        // Parent URL, parsed once and reused as the origin for the strategy checks below.
        let sitemapUrl: URL | undefined;

        if (source.type === 'url') {
            sitemapUrl = new URL(source.url);
            visitedSitemapUrls.add(sitemapUrl.toString());
            let retriesLeft = sitemapRetries + 1;

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Verify the sitemap URL actually returns sitemap XML/TXT (curl -I to check Content-Type)
2. Check the response isn't a soft-404 or bot-challenge HTML page
3. Serve sitemaps with correct Content-Type (application/xml, text/plain, application/x-gzip)
4. If content is correct, fix server MIME config so the essence matches a supported type
5. Skip/queue the source and continue with remaining sitemap sources in a try-catch

Example fix

// before
const urls = await parseSitemap([sitemapUrl]);

// after
try {
  const urls = await parseSitemap([sitemapUrl]);
} catch (err) {
  if (String(err).includes('Unsupported sitemap content type')) {
    log.warning(`skipping ${sitemapUrl}: not a sitemap`);
  } else throw err;
}

const res = await fetch(sitemapUrl, { method: 'HEAD' });
const ct = res.headers.get('content-type') ?? '';
const ok = /xml|gzip|text\/plain|octet-stream/.test(ct) || sitemapUrl.endsWith('.xml') || sitemapUrl.endsWith('.txt');
if (!ok) console.warn(`${sitemapUrl} is probably not a sitemap (${ct})`);

function looksLikeSitemapUrl(u: string): boolean {
  return /\.(xml|gz|txt)(\?|$)/.test(u);
}

try {
  urls = await parseSitemap([sitemapUrl]);
} catch (err) {
  if (String(err).includes('Unsupported sitemap content type')) urls = [];
  else throw err;
}

When it happens

Trigger: Calling parseSitemap()/SitemapUrlListLoader on a URL whose response Content-Type is something like 'application/json' or 'text/html' and whose path does not end in .xml, .gz, or .txt — e.g. an HTML error page or a REST endpoint instead of a sitemap.

Common situations: robots.txt pointing to a sitemap index that returns an HTML 404/soft-404 page; CDN serving HTML challenge pages; sitemap behind an endpoint that ignores Accept headers; typo'd sitemap URL; server sending an unlisted Content-Type like 'application/octet-stream'.

Related errors

• Could not parse cookie header string: ${cookieHeaderString}
• Unsupported content type: ${contentType}
• The "value" parameter must be a String, Buffer, ArrayBuffer,
• The request is not being processed (url: ${url})
• The `contentType` property is not available - `skipNavigatio

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/84fedf0a9be4d8e1. Report an issue: GitHub.