apify/crawlee · error · SessionError

Blocked by Cloudflare when processing ${url}

Error message

Blocked by Cloudflare when processing ${url}

What it means

During Cloudflare challenge handling, after waiting for the challenge to resolve, retryBlocked runs the user-supplied isBlockedCallback; if it reports the page is still blocked, a SessionError is thrown so the crawler rotates the session/proxy and retries with a fresh identity.

Source

Thrown at packages/playwright-crawler/src/internals/utils/playwright-utils.ts:700

        const isBlocked = await page.evaluate(() => {
            return document.querySelector('h1')?.textContent?.trim().includes('Sorry, you have been blocked');
        });
        return !!isBlocked;
    };

    options.isChallengeCallback ??= async () => {
        return await page.evaluate(async () => {
            // Cloudflare keeps reshuffling the wrapper elements between `.footer-inner` and `.ray-id`,
            // so only the stable outer classes are matched.
            return !!document.querySelector('.footer .footer-inner .ray-id');
        });
    };

    const retryBlocked = async () => {
        const isBlocked = await options.isBlockedCallback!(page).catch(() => false);

        if (isBlocked) {
            throw new SessionError(`Blocked by Cloudflare when processing ${url}`);
        }
    };

    // check if we ended up on the CF challenge page
    const isChallenge = async () => {
        return options.isChallengeCallback!(page).catch(() => false);
    };

    if (!(await isChallenge())) {
        await retryBlocked();
        return undefined;
    }

    const logLevel = options.verbose ? 'info' : 'debug';
    getLog()[logLevel](
        `Detected Cloudflare challenge at ${url}, trying to solve it. This can take up to ${10 + (options.sleepSecs ?? 10)} seconds.`,
    );

View on GitHub (pinned to dbe57fb09c)

Solutions

  1. Rotate to residential/mobile proxies via ProxyConfiguration — datacenter IPs are typically the root cause.
  2. Improve isBlockedCallback to distinguish challenge pages from soft blocks, or increase sleepSecs so the challenge fully resolves before the check.
  3. Reduce request rate and reuse sessions per domain to avoid IP reputation damage; let SessionError trigger automatic session rotation.

Example fix

// before
const crawler = new PlaywrightCrawler({
    proxyConfiguration: await ProxyConfiguration.create({ proxyUrls: ['http://datacenter-proxy:8080'] }),
    browserPoolOptions: { useFingerprints: true },
});
// after
const crawler = new PlaywrightCrawler({
    proxyConfiguration: await ProxyConfiguration.create({ groups: ['RESIDENTIAL'] }),
    browserPoolOptions: { useFingerprints: true, fingerprintOptions: { fingerprintGeneratorOptions: { browsers: ['chrome'] } } },
});
Defensive patterns

Strategy: retry

Validate before calling

// pre-flight proxy quality check
const isBlocked = await options.isBlockedCallback?.(page);
if (isBlocked) await session.retire(); // rotate before the crawler errors

Try / catch

try {
    await requestHandler(context);
} catch (err) {
    if (err instanceof SessionError && /Blocked by Cloudflare/.test(err.message)) {
        // crawler rotates session/proxy automatically; optionally back off
        await sleep(5000);
        throw err; // rethrow to trigger session rotation
    }
    throw err;
}

Prevention

When it happens

Trigger: handleCloudflareChallenge is active (blockedStatusCodes / Cloudflare handling enabled), the initial challenge check passes (not on challenge page), but options.isBlockedCallback(page) returns true — meaning the site still blocks the session even without a visible challenge.

Common situations: Datacenter proxy IPs flagged by Cloudflare; site-level IP rate limiting independent of the challenge; isBlockedCallback detecting bot-protection markers after solving the challenge.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/2e3c97080785c85f. Report an issue: GitHub.