apify/crawlee · warning · Error

Request timed out after ${this.#navigationTimeoutMillis / 10

Error message

Request timed out after ${this.#navigationTimeoutMillis / 1000} seconds.

What it means

handleRequestTimeout marks the session bad and throws `Request timed out after N seconds.` where N is navigationTimeoutMillis / 1000. It fires when the HTTP request (including redirects and body download) exceeds the configured navigation timeout, signaling the request should be retried or eventually failed.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:862

                this.#supportedMimeTypes.add(mimeType);
                continue;
            }

            try {
                const parsedType = contentTypeParser.parse(mimeType);
                this.#supportedMimeTypes.add(parsedType.type);
            } catch (err) {
                throw new Error(`Can not parse mime type ${mimeType} from "options.additionalMimeTypes".`);
            }
        }
    }

    /**
     * Handles timeout request
     */
    private handleRequestTimeout(session: ISession) {
        session.markBad();
        throw new Error(`Request timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`);
    }

    private abortDownloadOfBody(request: CrawleeRequest, response: Response) {
        const { status } = response;
        const { type } = parseContentTypeFromResponse(response);

        const isTransientContentType = status >= 500 || this.blockedStatusCodes.has(status);

        if (!this.#supportedMimeTypes.has(type) && !this.#supportedMimeTypes.has('*/*') && !isTransientContentType) {
            request.noRetry = true;
            throw new Error(
                `Resource ${request.url} served Content-Type ${type}, ` +
                    `but only ${Array.from(this.#supportedMimeTypes).join(', ')} are allowed. Skipping resource.`,
            );
        }
    }

    /**

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Increase `navigationTimeoutSecs` in HttpCrawler options to cover the slowest expected requests.
2. Check proxy performance; test the URL with curl through the same proxy to measure real latency.
3. Skip or special-case very large resources (filter by URL/content-length) instead of timing out.
4. Ensure request retries are configured so timeouts are retried rather than immediately failing.

Example fix

// before
new HttpCrawler({ navigationTimeoutSecs: 30 });

// after
new HttpCrawler({ navigationTimeoutSecs: 180, maxRequestRetries: 3 });

const start = Date.now();
await got(url, { timeout: { request: 60000 }, throwHttpErrors: false });
console.log(`Sample latency: ${Date.now() - start}ms — set navigationTimeoutSecs well above this`);

function isTimeoutError(err: unknown): err is Error {
  return err instanceof Error && /^Request timed out after \d+/.test(err.message);
}

try {
  await crawler.run(requests);
} catch (err) {
  if (err instanceof Error && err.message.startsWith('Request timed out after')) {
    log.warning(`Timeouts detected (${err.message}); consider raising navigationTimeoutSecs`);
  } else {
    throw err;
  }
}

When it happens

Trigger: A request exceeds `navigationTimeoutSecs` (default 60s) — slow target server, huge file download over a slow proxy, or a hung connection without response.

Common situations: Downloading large binaries (PDFs, images) through slow proxies; scraping extremely slow legacy sites; too-low navigationTimeoutSecs for heavy pages; proxy connection stalls.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Cannot fetch a request list from ${requestsFromUrl}: ${err}
• Cannot fetch a request list from ${requestsFromUrl}: ${err}
• Too many redirects (${maxRedirects}) while requesting ${curr
• Failed to load robots.txt from ${url}: HTTP ${response.statu
• The current SessionPool instance couldn't find a valid sessi

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/e9965904c9a6db44. Report an issue: GitHub.