{"record":{"id":"bcc04a27a734d86d","repo":"apify/crawlee","slug":"request-blocked-received-statuscode-status-co","errorCode":null,"errorMessage":"Request blocked - received ${statusCode} status code.","messagePattern":"Request blocked - received (.+?) status code\\.","errorType":"exception","errorClass":"SessionError","httpStatus":null,"severity":"error","filePath":"packages/basic-crawler/src/internals/basic-crawler.ts","lineNumber":2458,"sourceCode":"                    );\n                }\n\n                transaction.rollback();\n            }\n\n            // Unconditional: `failed` is a terminal state that the branch above never reaches.\n            transaction.dispose();\n        }\n    }\n\n    /**\n     * Handles blocked request\n     */\n    protected throwOnBlockedRequest(statusCode: number) {\n        if (this.retryOnBlocked) return;\n\n        if (this.blockedStatusCodes.has(statusCode)) {\n            throw new SessionError(`Request blocked - received ${statusCode} status code.`);\n        }\n    }\n\n    private async isAllowedBasedOnRobotsTxtFile(url: string): Promise<boolean> {\n        if (!this.#respectRobotsTxtFile) {\n            return true;\n        }\n\n        const robotsTxtFile = await this.getRobotsTxtFileForUrl(url);\n        const userAgent = typeof this.#respectRobotsTxtFile === 'object' ? this.#respectRobotsTxtFile?.userAgent : '*';\n\n        if (robotsTxtFile) {\n            const crawlDelay = robotsTxtFile.getCrawlDelay(userAgent);\n            if (crawlDelay !== undefined) {\n                this.applyCrawlDelay(url, crawlDelay);\n            }\n        }\n","sourceCodeStart":2440,"sourceCodeEnd":2476,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/basic-crawler/src/internals/basic-crawler.ts#L2440-L2476","documentation":"BasicCrawler detects blocked requests by HTTP status code. When the received status code is in the configured blockedStatusCodes set and retryOnBlocked is disabled, it throws a SessionError signaling the request was blocked (e.g. by anti-bot protection).","triggerScenarios":"A request returns a status code present in blockedStatusCodes (403, 429, etc.) while the retryOnBlocked option is false/unset, so the crawler fails fast instead of retrying with a new session.","commonSituations":"Scraping sites with Cloudflare/anti-bot protection that return 403; rate limiting returning 429; users disabling retryOnBlocked and then hitting bot detection.","solutions":["Set retryOnBlocked: true in the crawler options so blocked requests are retried with new sessions","Use residential/datacenter proxies or a proxy rotation to avoid blocks","Remove the blocking status code from blockedStatusCodes only if it is a false positive in your context","Reduce request rate / add delays to avoid triggering anti-bot measures"],"exampleFix":"// before\nconst crawler = new CheerioCrawler({ /* retryOnBlocked defaults to false */ });\n// after\nconst crawler = new CheerioCrawler({ retryOnBlocked: true });","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { await crawler.run(urls); } catch (e) { if (e instanceof SessionError && /Request blocked/.test(e.message)) { /* rotate proxy, increase concurrency delay, or re-enqueue */ } else throw e; }","preventionTips":["Enable retryOnBlocked: true in crawler options","Use proxy rotation and realistic request rates","Monitor 403/429 rates and rotate sessions proactively"],"tags":["network","anti-bot","session"],"backgroundTag":"request-blocked-status-code","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}