{"record":{"id":"2e3c97080785c85f","repo":"apify/crawlee","slug":"blocked-by-cloudflare-when-processing-url","errorCode":null,"errorMessage":"Blocked by Cloudflare when processing ${url}","messagePattern":"Blocked by Cloudflare when processing (.+?)","errorType":"exception","errorClass":"SessionError","httpStatus":null,"severity":"error","filePath":"packages/playwright-crawler/src/internals/utils/playwright-utils.ts","lineNumber":700,"sourceCode":"        const isBlocked = await page.evaluate(() => {\n            return document.querySelector('h1')?.textContent?.trim().includes('Sorry, you have been blocked');\n        });\n        return !!isBlocked;\n    };\n\n    options.isChallengeCallback ??= async () => {\n        return await page.evaluate(async () => {\n            // Cloudflare keeps reshuffling the wrapper elements between `.footer-inner` and `.ray-id`,\n            // so only the stable outer classes are matched.\n            return !!document.querySelector('.footer .footer-inner .ray-id');\n        });\n    };\n\n    const retryBlocked = async () => {\n        const isBlocked = await options.isBlockedCallback!(page).catch(() => false);\n\n        if (isBlocked) {\n            throw new SessionError(`Blocked by Cloudflare when processing ${url}`);\n        }\n    };\n\n    // check if we ended up on the CF challenge page\n    const isChallenge = async () => {\n        return options.isChallengeCallback!(page).catch(() => false);\n    };\n\n    if (!(await isChallenge())) {\n        await retryBlocked();\n        return undefined;\n    }\n\n    const logLevel = options.verbose ? 'info' : 'debug';\n    getLog()[logLevel](\n        `Detected Cloudflare challenge at ${url}, trying to solve it. This can take up to ${10 + (options.sleepSecs ?? 10)} seconds.`,\n    );\n","sourceCodeStart":682,"sourceCodeEnd":718,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L682-L718","documentation":"During Cloudflare challenge handling, after waiting for the challenge to resolve, retryBlocked runs the user-supplied isBlockedCallback; if it reports the page is still blocked, a SessionError is thrown so the crawler rotates the session/proxy and retries with a fresh identity.","triggerScenarios":"handleCloudflareChallenge is active (blockedStatusCodes / Cloudflare handling enabled), the initial challenge check passes (not on challenge page), but options.isBlockedCallback(page) returns true — meaning the site still blocks the session even without a visible challenge.","commonSituations":"Datacenter proxy IPs flagged by Cloudflare; site-level IP rate limiting independent of the challenge; isBlockedCallback detecting bot-protection markers after solving the challenge.","solutions":["Rotate to residential/mobile proxies via ProxyConfiguration — datacenter IPs are typically the root cause.","Improve isBlockedCallback to distinguish challenge pages from soft blocks, or increase sleepSecs so the challenge fully resolves before the check.","Reduce request rate and reuse sessions per domain to avoid IP reputation damage; let SessionError trigger automatic session rotation."],"exampleFix":"// before\nconst crawler = new PlaywrightCrawler({\n    proxyConfiguration: await ProxyConfiguration.create({ proxyUrls: ['http://datacenter-proxy:8080'] }),\n    browserPoolOptions: { useFingerprints: true },\n});\n// after\nconst crawler = new PlaywrightCrawler({\n    proxyConfiguration: await ProxyConfiguration.create({ groups: ['RESIDENTIAL'] }),\n    browserPoolOptions: { useFingerprints: true, fingerprintOptions: { fingerprintGeneratorOptions: { browsers: ['chrome'] } } },\n});","handlingStrategy":"retry","validationCode":"// pre-flight proxy quality check\nconst isBlocked = await options.isBlockedCallback?.(page);\nif (isBlocked) await session.retire(); // rotate before the crawler errors","typeGuard":null,"tryCatchPattern":"try {\n    await requestHandler(context);\n} catch (err) {\n    if (err instanceof SessionError && /Blocked by Cloudflare/.test(err.message)) {\n        // crawler rotates session/proxy automatically; optionally back off\n        await sleep(5000);\n        throw err; // rethrow to trigger session rotation\n    }\n    throw err;\n}","preventionTips":["Use residential proxies for Cloudflare-protected sites.","Keep useFingerprints enabled and rates modest.","Retire suspicious sessions proactively from isBlockedCallback."],"tags":["cloudflare","session-rotation","proxy","anti-bot","crawlee"],"backgroundTag":"blocked-by-cloudflare","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}