{"record":{"id":"519d068a0a5a8cf3","repo":"apify/crawlee","slug":"dynamic-blocked-request-message-returned-by-isre","errorCode":null,"errorMessage":"<dynamic: blocked-request message returned by isRequestBlocked>","messagePattern":"<dynamic: blocked-request message returned by isRequestBlocked>","errorType":"exception","errorClass":"SessionError","httpStatus":null,"severity":"warning","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":668,"sourceCode":"        }\n\n        return {\n            get json() {\n                if (contentType.type !== APPLICATION_JSON_MIME_TYPE) return null;\n                const jsonString = parsed.body!.toString(contentType.encoding);\n                return JSON.parse(jsonString);\n            },\n            waitForSelector,\n            parseWithCheerio,\n            contentType,\n            body: parsed.body,\n        };\n    }\n\n    private async handleBlockedRequestByContent(crawlingContext: InternalHttpCrawlingContext): Promise<{}> {\n        if (this.retryOnBlocked) {\n            const error = await this.isRequestBlocked(crawlingContext);\n            if (error) throw new SessionError(error);\n        }\n        return {};\n    }\n\n    protected async isRequestBlocked(crawlingContext: InternalHttpCrawlingContext): Promise<string | false> {\n        if (HTML_AND_XML_MIME_TYPES.includes(crawlingContext.contentType.type)) {\n            const $ = await crawlingContext.parseWithCheerio();\n\n            const foundSelectors = RETRY_CSS_SELECTORS.filter((selector) => $(selector).length > 0);\n\n            if (foundSelectors.length > 0) {\n                return `Found selectors: ${foundSelectors.join(', ')}`;\n            }\n        }\n\n        if (this.blockedStatusCodes.has(crawlingContext.response.status!)) {\n            return `Blocked by status code ${crawlingContext.response.status}`;\n        }","sourceCodeStart":650,"sourceCodeEnd":686,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L650-L686","documentation":"When `retryOnBlocked` is enabled, HttpCrawler calls the user-overridable `isRequestBlocked()` after each response; if it returns a truthy message, that message is thrown wrapped in a SessionError. SessionError signals that the current session (proxy IP / cookies) appears blocked and the request should be retried with a different session. The message text is produced by the block-detection logic, not fixed by the library.","triggerScenarios":"A response whose content matches blocking heuristics (default: status codes in blockedStatusCodes like 403/429, or error-page content) while `retryOnBlocked: true`; or a custom `isRequestBlocked` override returning a string.","commonSituations":"Target site rate-limits or CAPTCHAs your proxy IPs; WAF returns a 200 page with 'Access denied' content; misconfigured custom isRequestBlocked marking valid pages as blocked; scraping without sessions configured so retries have nothing to rotate.","solutions":["Configure `sessionPoolOptions` and `proxyConfiguration` so blocked sessions are retired and retried with fresh IPs/cookies.","Increase `maxRequestRetries` and add backoff so rate limits clear between attempts.","Slow down request rate (maxConcurrency, maxRequestsPerMinute) or rotate user agents/headers to avoid triggering blocks.","Review/adjust your `isRequestBlocked` override if legitimate responses are being flagged."],"exampleFix":"// before\nnew HttpCrawler({ retryOnBlocked: true });\n\n// after\nnew HttpCrawler({\n  retryOnBlocked: true,\n  useSessionPool: true,\n  persistCookiesPerSession: true,\n  proxyConfiguration: await ProxyConfiguration.createProxyConfiguration(),\n});","handlingStrategy":"retry","validationCode":"const res = await got(url, { throwHttpErrors: false, headers });\nif (res.statusCode === 403 || res.statusCode === 429) {\n  log.warning('Likely blocked; ensure sessionPool + proxyConfiguration are configured');\n}","typeGuard":"function isSessionError(err: unknown): err is SessionError {\n  return err instanceof SessionError;\n}","tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  if (err instanceof SessionError) {\n    log.warning(`Blocked session detected: ${err.message}`); // crawler already retries with new session\n  } else {\n    throw err;\n  }\n}","preventionTips":["Always pair retryOnBlocked: true with useSessionPool: true and a proxyConfiguration.","Set persistCookiesPerSession so session rotation carries consistent cookies.","Tune maxRequestRetries and request throttling (maxRequestsPerMinute) for aggressively protected sites.","Keep isRequestBlocked overrides narrow to avoid flagging valid pages."],"tags":["session","blocked","anti-scraping","retry"],"backgroundTag":"session-blocked","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}