{"record":{"id":"d9c3a9ae22ffafa5","repo":"apify/crawlee","slug":"navigation-timed-out-after-this-navigationtimeo-d9c3a9","errorCode":null,"errorMessage":"Navigation timed out after ${this.#navigationTimeoutMillis / 1000} seconds.","messagePattern":"Navigation timed out after (.+?) seconds\\.","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":474,"sourceCode":"        // When navigation is skipped, `prepareHttpRequest` has already installed throwing getters for\n        // the response-derived members, so the guarded action is bypassed and the context left untouched.\n        const skipGuard = <Ctx extends CrawlingContext, Ext>(\n            action: (ctx: Ctx) => Awaitable<void | Ext>,\n        ): ContextMiddleware<Ctx, Ext> => ({\n            action: async (ctx) => (ctx.request.skipNavigation ? {} : ((await action(ctx)) ?? {})) as Ext,\n        });\n\n        // A single navigation window covers the pre-navigation hooks, the navigation, and the post-navigation\n        // hooks: the whole phase shares one `navigationTimeoutSecs` budget, so a slow hook eats into the same\n        // window the navigation uses instead of each step being timed on its own.\n        const navigationTimedOut = `Navigation timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`;\n        const windowGuard = <Ctx extends CrawlingContext, Ext>(\n            step: (ctx: Ctx) => Awaitable<void | Ext>,\n        ): ContextMiddleware<Ctx, Ext> =>\n            skipGuard(async (ctx: Ctx) => {\n                const remaining = remainingNavigationWindowMillis(ctx, this.#navigationTimeoutMillis);\n                if (remaining <= 0) {\n                    throw new TimeoutError(navigationTimedOut);\n                }\n                return addTimeoutToPromise(async () => step(ctx), remaining, navigationTimedOut);\n            });\n\n        let pipeline = ContextPipeline.create<CrawlingContext>().compose({\n            action: this.prepareHttpRequest.bind(this),\n        });\n\n        for (const hook of this.#preNavigationHooks) {\n            pipeline = pipeline.compose(windowGuard(hook));\n        }\n\n        let pipelineWithNavigation = pipeline.compose(skipGuard(this.makeHttpRequest.bind(this)));\n\n        for (const hook of this.#postNavigationHooks) {\n            pipelineWithNavigation = pipelineWithNavigation.compose(windowGuard(hook));\n        }\n","sourceCodeStart":456,"sourceCodeEnd":492,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L456-L492","documentation":"TimeoutError thrown by the crawler's navigation window guard. The http-crawler enforces a per-request navigation window (`#navigationTimeoutMillis`); when a request handler step is about to run with no time left, or when the step exceeds the remaining window, the crawler aborts it with this message so the request can be retried or marked failed.","triggerScenarios":"A request reaches `windowGuard` after `remainingNavigationWindowMillis` is <= 0 (all navigation budget already consumed by earlier steps/hooks), or the handler step inside `addTimeoutToPromise` runs longer than the remaining window (e.g. slow userHandler, slow parsing, hangs waiting on I/O).","commonSituations":"Large userHandler doing heavy cheerio/DOM parsing or slow response body reads; long post-navigation hooks eating the window; setting `navigationTimeoutSecs` too low in crawler options; slow targets causing retries that consume per-attempt time.","solutions":["Increase `navigationTimeoutSecs` in the crawler options (e.g. new HttpCrawler({ navigationTimeoutSecs: 120 })).","Move expensive work out of the request handler or split it; keep the handler inside the navigation window.","Call `requestHandlerTimeoutSecs` / hook timeouts appropriately and avoid blocking awaits (slow fetches, sleep loops) in handlers.","If the target is just slow, use `keepAlive`/retry configuration and make sure slow body reads aren't hitting the same shared window."],"exampleFix":"// before\nconst crawler = new HttpCrawler({ navigationTimeoutSecs: 30, async requestHandler(ctx) { await parseHugePage(ctx); } });\n// after\nconst crawler = new HttpCrawler({ navigationTimeoutSecs: 120, requestHandlerTimeoutSecs: 120, async requestHandler(ctx) { await parsePage(ctx); } });","handlingStrategy":"retry","validationCode":"// estimate budget before heavy work\nconst remaining = Date.now() - ctx.request.userData._navStartMs ?? 0; // track via preNavigationHook\nif (remaining > 0.8 * 30_000) { await heavyWork(ctx); }","typeGuard":null,"tryCatchPattern":"try { await crawler.run(requests); } catch (err) { if (err instanceof TimeoutError) { /* increase navigationTimeoutSecs and rerun failed requests */ } throw err; }","preventionTips":["Set navigationTimeoutSecs generously relative to slowest target.","Keep requestHandler work bounded; offload heavy CPU work.","Track request timing in userData via hooks to detect near-expiry requests.","Monitor TimeoutError rates in failedRequestHandler."],"tags":["timeout","navigation","http-crawler"],"backgroundTag":"navigation-timeout","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}