apify/crawlee · warning · Error

Resource ${request.url} served Content-Type ${type}, but onl

Error message

Resource ${request.url} served Content-Type ${type}, but only ${Array.from(this.#supportedMimeTypes).join(', ')} are allowed. Skipping resource.

What it means

abortDownloadOfBody runs when the response must not be processed further: if the Content-Type is not in the crawler's supported MIME types (default text/html, application/json, plus additionalMimeTypes, unless */* is allowed) and the status is not transient (>=500 or a blocked status), it sets request.noRetry = true and throws this error. The request is skipped rather than retried, since the content will never be usable.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:873

    }

    /**
     * Handles timeout request
     */
    private handleRequestTimeout(session: ISession) {
        session.markBad();
        throw new Error(`Request timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`);
    }

    private abortDownloadOfBody(request: CrawleeRequest, response: Response) {
        const { status } = response;
        const { type } = parseContentTypeFromResponse(response);

        const isTransientContentType = status >= 500 || this.blockedStatusCodes.has(status);

        if (!this.#supportedMimeTypes.has(type) && !this.#supportedMimeTypes.has('*/*') && !isTransientContentType) {
            request.noRetry = true;
            throw new Error(
                `Resource ${request.url} served Content-Type ${type}, ` +
                    `but only ${Array.from(this.#supportedMimeTypes).join(', ')} are allowed. Skipping resource.`,
            );
        }
    }

    /**
     * @internal wraps public utility for mocking purposes
     */
    private requestAsBrowser = async (options: Dictionary<any>, session: ISession) => {
        const opts = processHttpRequestOptions({
            ...(options as any),
            responseType: 'text',
        });

        // When saveResponseCookies is false, the response cookies must not mutate the
        // session jar. Reads still go through the session (so session.setCookie() in pre-nav
        // hooks keeps working) but a per-request clone is passed in so writes are discarded.

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Add the missing type to `additionalMimeTypes`, e.g. ['application/pdf', 'image/png'], if you actually want to handle it.
2. Filter requests before enqueueing: skip URLs ending in media/PDF extensions or use `request.userData`/preNavigationHooks checks.
3. If you want every content type, note that wildcards are checked via '*/*' support in extendSupportedMimeTypes — add appropriate handling, or ignore the error as a benign skip.
4. Set request.noRetry awareness in failedRequestHandler so skipped resources are not logged as real failures.

Example fix

// before
new HttpCrawler({}); // chokes on application/pdf links

// after
await crawler.addRequests(urls.filter((u) => !/\.(pdf|zip|mp4|png|jpe?g)$/i.test(u)));
// or, to actually process them:
new HttpCrawler({ additionalMimeTypes: ['application/pdf'] });

const MEDIA_EXT = /\.(pdf|zip|rar|mp[34]|avi|png|jpe?g|gif|svg|woff2?|ttf|exe|dmg)$/i;
const allowed = requests.filter((r) => !MEDIA_EXT.test(new URL(r.url).pathname));
await crawler.addRequests(allowed);

function isProbablyDocumentUrl(url: string): boolean {
  return /\.(pdf|zip|mp4|png|jpe?g|docx?|xlsx?)$/i.test(new URL(url).pathname);
}

try {
  await crawler.run(requests);
} catch (err) {
  if (err instanceof Error && err.message.includes('are allowed. Skipping resource.')) {
    log.debug(`Skipped unsupported content type (expected): ${err.message.slice(0, 120)}`);
    return;
  }
  throw err;
}

When it happens

Trigger: Crawling links that return images, videos, ZIPs, or other unsupported types (image/png, video/mp4, application/pdf, application/octet-stream) without adding them to additionalMimeTypes; hitting non-HTML sitemaps or feeds without XML support.

Common situations: Following <a> links to PDFs or media files; crawling sites that serve application/octet-stream for downloads; forgetting to add text/csv or application/xml when scraping data exports; crawlers picking up direct file URLs from listing pages.

Related errors

• ${status} - Error status code was set by user.
• Can not parse mime type ${mimeType} from "options.additional
• The `minConcurrency`/`maxConcurrency`/`initialConcurrency`/`
• The `requestManager` option cannot be used in conjunction wi
• Cannot decide what to purge before running again: `sameDomai

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/f33863e06ca7800d. Report an issue: GitHub.