{"record":{"id":"f33863e06ca7800d","repo":"apify/crawlee","slug":"resource-request-url-served-content-type-type","errorCode":null,"errorMessage":"Resource ${request.url} served Content-Type ${type}, but only ${Array.from(this.#supportedMimeTypes).join(', ')} are allowed. Skipping resource.","messagePattern":"Resource (.+?) served Content-Type (.+?), but only (.+?) are allowed\\. Skipping resource\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":873,"sourceCode":"    }\n\n    /**\n     * Handles timeout request\n     */\n    private handleRequestTimeout(session: ISession) {\n        session.markBad();\n        throw new Error(`Request timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`);\n    }\n\n    private abortDownloadOfBody(request: CrawleeRequest, response: Response) {\n        const { status } = response;\n        const { type } = parseContentTypeFromResponse(response);\n\n        const isTransientContentType = status >= 500 || this.blockedStatusCodes.has(status);\n\n        if (!this.#supportedMimeTypes.has(type) && !this.#supportedMimeTypes.has('*/*') && !isTransientContentType) {\n            request.noRetry = true;\n            throw new Error(\n                `Resource ${request.url} served Content-Type ${type}, ` +\n                    `but only ${Array.from(this.#supportedMimeTypes).join(', ')} are allowed. Skipping resource.`,\n            );\n        }\n    }\n\n    /**\n     * @internal wraps public utility for mocking purposes\n     */\n    private requestAsBrowser = async (options: Dictionary<any>, session: ISession) => {\n        const opts = processHttpRequestOptions({\n            ...(options as any),\n            responseType: 'text',\n        });\n\n        // When saveResponseCookies is false, the response cookies must not mutate the\n        // session jar. Reads still go through the session (so session.setCookie() in pre-nav\n        // hooks keeps working) but a per-request clone is passed in so writes are discarded.","sourceCodeStart":855,"sourceCodeEnd":891,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L855-L891","documentation":"abortDownloadOfBody runs when the response must not be processed further: if the Content-Type is not in the crawler's supported MIME types (default text/html, application/json, plus additionalMimeTypes, unless */* is allowed) and the status is not transient (>=500 or a blocked status), it sets request.noRetry = true and throws this error. The request is skipped rather than retried, since the content will never be usable.","triggerScenarios":"Crawling links that return images, videos, ZIPs, or other unsupported types (image/png, video/mp4, application/pdf, application/octet-stream) without adding them to additionalMimeTypes; hitting non-HTML sitemaps or feeds without XML support.","commonSituations":"Following <a> links to PDFs or media files; crawling sites that serve application/octet-stream for downloads; forgetting to add text/csv or application/xml when scraping data exports; crawlers picking up direct file URLs from listing pages.","solutions":["Add the missing type to `additionalMimeTypes`, e.g. ['application/pdf', 'image/png'], if you actually want to handle it.","Filter requests before enqueueing: skip URLs ending in media/PDF extensions or use `request.userData`/preNavigationHooks checks.","If you want every content type, note that wildcards are checked via '*/*' support in extendSupportedMimeTypes — add appropriate handling, or ignore the error as a benign skip.","Set request.noRetry awareness in failedRequestHandler so skipped resources are not logged as real failures."],"exampleFix":"// before\nnew HttpCrawler({}); // chokes on application/pdf links\n\n// after\nawait crawler.addRequests(urls.filter((u) => !/\\.(pdf|zip|mp4|png|jpe?g)$/i.test(u)));\n// or, to actually process them:\nnew HttpCrawler({ additionalMimeTypes: ['application/pdf'] });","handlingStrategy":"validation","validationCode":"const MEDIA_EXT = /\\.(pdf|zip|rar|mp[34]|avi|png|jpe?g|gif|svg|woff2?|ttf|exe|dmg)$/i;\nconst allowed = requests.filter((r) => !MEDIA_EXT.test(new URL(r.url).pathname));\nawait crawler.addRequests(allowed);","typeGuard":"function isProbablyDocumentUrl(url: string): boolean {\n  return /\\.(pdf|zip|mp4|png|jpe?g|docx?|xlsx?)$/i.test(new URL(url).pathname);\n}","tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('are allowed. Skipping resource.')) {\n    log.debug(`Skipped unsupported content type (expected): ${err.message.slice(0, 120)}`);\n    return;\n  }\n  throw err;\n}","preventionTips":["Add every content type you intend to parse to additionalMimeTypes.","Filter media/file-extension URLs before enqueueing with addRequests.","Note request.noRetry is set — these skips are by design; handle them in failedRequestHandler as informational.","If the site serves everything as application/octet-stream, inspect Content-Disposition to decide handling."],"tags":["mime-type","content-type","http","configuration"],"backgroundTag":"unsupported-content-type","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}