apify/crawlee · error · SessionError

Request blocked - received ${statusCode} status code.

Error message

Request blocked - received ${statusCode} status code.

What it means

BasicCrawler detects blocked requests by HTTP status code. When the received status code is in the configured blockedStatusCodes set and retryOnBlocked is disabled, it throws a SessionError signaling the request was blocked (e.g. by anti-bot protection).

Source

Thrown at packages/basic-crawler/src/internals/basic-crawler.ts:2458

                    );
                }

                transaction.rollback();
            }

            // Unconditional: `failed` is a terminal state that the branch above never reaches.
            transaction.dispose();
        }
    }

    /**
     * Handles blocked request
     */
    protected throwOnBlockedRequest(statusCode: number) {
        if (this.retryOnBlocked) return;

        if (this.blockedStatusCodes.has(statusCode)) {
            throw new SessionError(`Request blocked - received ${statusCode} status code.`);
        }
    }

    private async isAllowedBasedOnRobotsTxtFile(url: string): Promise<boolean> {
        if (!this.#respectRobotsTxtFile) {
            return true;
        }

        const robotsTxtFile = await this.getRobotsTxtFileForUrl(url);
        const userAgent = typeof this.#respectRobotsTxtFile === 'object' ? this.#respectRobotsTxtFile?.userAgent : '*';

        if (robotsTxtFile) {
            const crawlDelay = robotsTxtFile.getCrawlDelay(userAgent);
            if (crawlDelay !== undefined) {
                this.applyCrawlDelay(url, crawlDelay);
            }
        }

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Set retryOnBlocked: true in the crawler options so blocked requests are retried with new sessions
2. Use residential/datacenter proxies or a proxy rotation to avoid blocks
3. Remove the blocking status code from blockedStatusCodes only if it is a false positive in your context
4. Reduce request rate / add delays to avoid triggering anti-bot measures

Example fix

// before
const crawler = new CheerioCrawler({ /* retryOnBlocked defaults to false */ });
// after
const crawler = new CheerioCrawler({ retryOnBlocked: true });

try { await crawler.run(urls); } catch (e) { if (e instanceof SessionError && /Request blocked/.test(e.message)) { /* rotate proxy, increase concurrency delay, or re-enqueue */ } else throw e; }

When it happens

Trigger: A request returns a status code present in blockedStatusCodes (403, 429, etc.) while the retryOnBlocked option is false/unset, so the crawler fails fast instead of retrying with a new session.

Common situations: Scraping sites with Cloudflare/anti-bot protection that return 403; rate limiting returning 429; users disabling retryOnBlocked and then hitting bot detection.

Related errors

• ${error} (possible values: 'Cloudflare challenge failed, fou
• ${this.getMessageFromError(error)}
• <dynamic: message extracted from proxied error via getMessag
• The current SessionPool instance couldn't find a valid sessi
• Failed to resolve the remote browser endpoint.

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/bcc04a27a734d86d. Report an issue: GitHub.