{"record":{"id":"e9965904c9a6db44","repo":"apify/crawlee","slug":"request-timed-out-after-this-navigationtimeoutm","errorCode":null,"errorMessage":"Request timed out after ${this.#navigationTimeoutMillis / 1000} seconds.","messagePattern":"Request timed out after (.+?) seconds\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":862,"sourceCode":"                this.#supportedMimeTypes.add(mimeType);\n                continue;\n            }\n\n            try {\n                const parsedType = contentTypeParser.parse(mimeType);\n                this.#supportedMimeTypes.add(parsedType.type);\n            } catch (err) {\n                throw new Error(`Can not parse mime type ${mimeType} from \"options.additionalMimeTypes\".`);\n            }\n        }\n    }\n\n    /**\n     * Handles timeout request\n     */\n    private handleRequestTimeout(session: ISession) {\n        session.markBad();\n        throw new Error(`Request timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`);\n    }\n\n    private abortDownloadOfBody(request: CrawleeRequest, response: Response) {\n        const { status } = response;\n        const { type } = parseContentTypeFromResponse(response);\n\n        const isTransientContentType = status >= 500 || this.blockedStatusCodes.has(status);\n\n        if (!this.#supportedMimeTypes.has(type) && !this.#supportedMimeTypes.has('*/*') && !isTransientContentType) {\n            request.noRetry = true;\n            throw new Error(\n                `Resource ${request.url} served Content-Type ${type}, ` +\n                    `but only ${Array.from(this.#supportedMimeTypes).join(', ')} are allowed. Skipping resource.`,\n            );\n        }\n    }\n\n    /**","sourceCodeStart":844,"sourceCodeEnd":880,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L844-L880","documentation":"handleRequestTimeout marks the session bad and throws `Request timed out after N seconds.` where N is navigationTimeoutMillis / 1000. It fires when the HTTP request (including redirects and body download) exceeds the configured navigation timeout, signaling the request should be retried or eventually failed.","triggerScenarios":"A request exceeds `navigationTimeoutSecs` (default 60s) — slow target server, huge file download over a slow proxy, or a hung connection without response.","commonSituations":"Downloading large binaries (PDFs, images) through slow proxies; scraping extremely slow legacy sites; too-low navigationTimeoutSecs for heavy pages; proxy connection stalls.","solutions":["Increase `navigationTimeoutSecs` in HttpCrawler options to cover the slowest expected requests.","Check proxy performance; test the URL with curl through the same proxy to measure real latency.","Skip or special-case very large resources (filter by URL/content-length) instead of timing out.","Ensure request retries are configured so timeouts are retried rather than immediately failing."],"exampleFix":"// before\nnew HttpCrawler({ navigationTimeoutSecs: 30 });\n\n// after\nnew HttpCrawler({ navigationTimeoutSecs: 180, maxRequestRetries: 3 });","handlingStrategy":"retry","validationCode":"const start = Date.now();\nawait got(url, { timeout: { request: 60000 }, throwHttpErrors: false });\nconsole.log(`Sample latency: ${Date.now() - start}ms — set navigationTimeoutSecs well above this`);","typeGuard":"function isTimeoutError(err: unknown): err is Error {\n  return err instanceof Error && /^Request timed out after \\d+/.test(err.message);\n}","tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith('Request timed out after')) {\n    log.warning(`Timeouts detected (${err.message}); consider raising navigationTimeoutSecs`);\n  } else {\n    throw err;\n  }\n}","preventionTips":["Benchmark the slowest target URLs and set navigationTimeoutSecs with headroom.","Use maxRequestRetries so one-off slow responses do not fail the crawl.","Watch proxy latency — stalls through proxies are the most common timeout cause.","Avoid downloading huge binaries with HttpCrawler; use direct downloads or increase the timeout."],"tags":["timeout","network","http","performance"],"backgroundTag":"request-timeout","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}