apify/crawlee · warning · SessionError

${error} (possible values: 'Cloudflare challenge failed, fou

Error message

${error} (possible values: 'Cloudflare challenge failed, found selectors: ${...}', 'Found selectors: ${...}', 'Received blocked status code: ${statusCode}')

What it means

`handleBlockedRequestByContent()` asks `isRequestBlocked()` whether the loaded page looks like an anti-bot block (Cloudflare challenge markers, known block selectors, or a blocked HTTP status code). When it returns a description and `retryOnBlocked` is enabled, the crawler throws a `SessionError` carrying that description so the session is retired and the request retried with a fresh session.

Source

Thrown at packages/browser-crawler/src/internals/browser-crawler.ts:769

     */
    protected override async runRequestHandler(crawlingContext: ExtendedContext): Promise<void> {
        try {
            await super.runRequestHandler(crawlingContext);
        } finally {
            if (!crawlingContext.request.skipNavigation) {
                try {
                    await this.persistCookiesFromPage(crawlingContext);
                } catch {
                    // Page may already be closed on some failure paths; ignore.
                }
            }
        }
    }

    private async handleBlockedRequestByContent(crawlingContext: BrowserCrawlingContext<Page, Response>) {
        if (this.retryOnBlocked) {
            const error = await this.isRequestBlocked(crawlingContext);
            if (error) throw new SessionError(error);
        }

        return {};
    }

    private async restoreRequestState(crawlingContext: CrawlingContext) {
        crawlingContext.request.state = RequestState.REQUEST_HANDLER;
        return {};
    }

    private async applyCookies(
        { session, request, page }: BrowserCrawlingContext<Page, Response>,
        preHooksCookies: string,
        postHooksCookies: string,
    ) {
        const sessionCookie = session
            ? (await session.cookieJar.getCookies(request.url)).map(toughCookieToBrowserPoolCookie)
            : [];

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Let the crawler retry automatically with a fresh session (the `SessionError` marks the session bad) and increase `maxRequestRetries` if blocks are transient
2. Improve the setup: use higher-quality residential proxies (`proxyConfiguration`), keep `useSessionPool` enabled, and set realistic browser fingerprints
3. Lower request concurrency and add delays so the domain is not hammered (rate-limit-triggered blocks)
4. Check `isRequestBlocked`/`blockSelector` configuration if you customized blocked-status codes — your site may be returning an unexpected status that now counts as blocked

Example fix

// before
const crawler = new PlaywrightCrawler({ /* default settings, no proxies */ });
// after
const crawler = new PlaywrightCrawler({
    retryOnBlocked: true,
    useSessionPool: true,
    maxConcurrency: 5,
    proxyConfiguration: await ProxyConfiguration.create({
        proxyUrls: ['http://residential-proxy:8000'],
    }),
});

// Detect the block yourself before/alongside the crawler so you can react (rotate session/proxy):
const html = await page.content();
const blocked = await crawler.isRequestBlocked?.({ page, response, request }) ??
    /Just a moment|Attention Required/i.test(html);

function isSessionError(e: unknown): e is SessionError {
    return e instanceof SessionError || (e instanceof Error && /blocked|Cloudflare|blocked status code/i.test(e.message));
}

crawler.failedRequestHandler = async ({ request, error }) => {
    if (error instanceof SessionError && /blocked/i.test(error.message)) {
        log.warning(`Blocked after all retries: ${request.url} — ${error.message}`);
        // persist URL for a later rerun with fresh proxies
    }
};

When it happens

Trigger: `retryOnBlocked: true` (or default) and the loaded page content matches block heuristics: Cloudflare challenge selectors present, known captcha/block selectors found in the DOM, or the response status code is on the blocked list (e.g. 403/429) for a target behind Cloudflare, PerimeterX, or similar protection.

Common situations: Scraping Cloudflare-protected sites with a headless browser that fails the challenge; a datacenter proxy IP range that is widely blocked; sessions whose cookies/user-agent fingerprint got flagged; after the target site upgrades its bot protection and old selectors start matching.

Related errors

• Request blocked - received ${statusCode} status code.
• Blocked by Cloudflare when processing ${url}
• The current SessionPool instance couldn't find a valid sessi
• ${this.getMessageFromError(error)}
• Cannot add session with id '${id}' as it already exists in t

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/f3de9b68c575f350. Report an issue: GitHub.