apify/crawlee · warning · SessionError

<dynamic: blocked-request message returned by isRequestBlock

Error message

<dynamic: blocked-request message returned by isRequestBlocked>

What it means

When `retryOnBlocked` is enabled, HttpCrawler calls the user-overridable `isRequestBlocked()` after each response; if it returns a truthy message, that message is thrown wrapped in a SessionError. SessionError signals that the current session (proxy IP / cookies) appears blocked and the request should be retried with a different session. The message text is produced by the block-detection logic, not fixed by the library.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:668

        }

        return {
            get json() {
                if (contentType.type !== APPLICATION_JSON_MIME_TYPE) return null;
                const jsonString = parsed.body!.toString(contentType.encoding);
                return JSON.parse(jsonString);
            },
            waitForSelector,
            parseWithCheerio,
            contentType,
            body: parsed.body,
        };
    }

    private async handleBlockedRequestByContent(crawlingContext: InternalHttpCrawlingContext): Promise<{}> {
        if (this.retryOnBlocked) {
            const error = await this.isRequestBlocked(crawlingContext);
            if (error) throw new SessionError(error);
        }
        return {};
    }

    protected async isRequestBlocked(crawlingContext: InternalHttpCrawlingContext): Promise<string | false> {
        if (HTML_AND_XML_MIME_TYPES.includes(crawlingContext.contentType.type)) {
            const $ = await crawlingContext.parseWithCheerio();

            const foundSelectors = RETRY_CSS_SELECTORS.filter((selector) => $(selector).length > 0);

            if (foundSelectors.length > 0) {
                return `Found selectors: ${foundSelectors.join(', ')}`;
            }
        }

        if (this.blockedStatusCodes.has(crawlingContext.response.status!)) {
            return `Blocked by status code ${crawlingContext.response.status}`;
        }

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Configure `sessionPoolOptions` and `proxyConfiguration` so blocked sessions are retired and retried with fresh IPs/cookies.
2. Increase `maxRequestRetries` and add backoff so rate limits clear between attempts.
3. Slow down request rate (maxConcurrency, maxRequestsPerMinute) or rotate user agents/headers to avoid triggering blocks.
4. Review/adjust your `isRequestBlocked` override if legitimate responses are being flagged.

Example fix

// before
new HttpCrawler({ retryOnBlocked: true });

// after
new HttpCrawler({
  retryOnBlocked: true,
  useSessionPool: true,
  persistCookiesPerSession: true,
  proxyConfiguration: await ProxyConfiguration.createProxyConfiguration(),
});

const res = await got(url, { throwHttpErrors: false, headers });
if (res.statusCode === 403 || res.statusCode === 429) {
  log.warning('Likely blocked; ensure sessionPool + proxyConfiguration are configured');
}

function isSessionError(err: unknown): err is SessionError {
  return err instanceof SessionError;
}

try {
  await crawler.run(requests);
} catch (err) {
  if (err instanceof SessionError) {
    log.warning(`Blocked session detected: ${err.message}`); // crawler already retries with new session
  } else {
    throw err;
  }
}

When it happens

Trigger: A response whose content matches blocking heuristics (default: status codes in blockedStatusCodes like 403/429, or error-page content) while `retryOnBlocked: true`; or a custom `isRequestBlocked` override returning a string.

Common situations: Target site rate-limits or CAPTCHAs your proxy IPs; WAF returns a 200 page with 'Access denied' content; misconfigured custom isRequestBlocked marking valid pages as blocked; scraping without sessions configured so retries have nothing to rotate.

Related errors

• <dynamic: message extracted from proxied error via getMessag
• The current SessionPool instance couldn't find a valid sessi
• Request blocked - received ${statusCode} status code.
• ${error} (possible values: 'Cloudflare challenge failed, fou
• ${this.getMessageFromError(error)}

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/519d068a0a5a8cf3. Report an issue: GitHub.