apify/crawlee · warning · ContextPipelineInterruptedError

Skipping request ${request.url} as disallowed by robots.txt

Error message

Skipping request ${request.url} as disallowed by robots.txt

What it means

When robots.txt analysis (robotsTxtFile feature) marks a URL as disallowed for the crawler's user agent, the crawler refuses to fetch it, records it as a skipped request, and interrupts the context pipeline with ContextPipelineInterruptedError. The request is marked noRetry so it is not retried.

Source

Thrown at packages/basic-crawler/src/internals/basic-crawler.ts:1463

            .compose({ action: this.checkRobotsTxt.bind(this) })
            .compose({ action: (context) => this.createBaseContext(context) })
            .compose({ action: this.resolveSession.bind(this) })
            .compose({ action: this.createContextHelpers.bind(this) });
    }

    private async checkRobotsTxt({ request }: { request: Request }) {
        if (!(await this.isAllowedBasedOnRobotsTxtFile(request.url))) {
            this.log.warning(
                `Skipping request ${request.url} (${request.id}) because it is disallowed based on robots.txt`,
            );
            request.state = RequestState.SKIPPED;
            request.noRetry = true;
            await this.#handleSkippedRequest({
                request,
                reason: 'robotsTxt',
            });

            throw new ContextPipelineInterruptedError(`Skipping request ${request.url} as disallowed by robots.txt`);
        }

        return {};
    }

    /**
     * Builds the subclass-specific context pipeline that transforms a `CrawlingContext` into the crawler's target context type.
     * Subclasses should override this to add their own pipeline stages.
     */
    protected buildContextPipeline(): ContextPipeline<CrawlingContext, CrawlingContext> {
        return ContextPipeline.create<CrawlingContext>();
    }

    private createBaseContext(context: PendingCrawlingContext) {
        const deferredCleanup: (() => Promise<unknown>)[] = [];

        return {
            id: cryptoRandomObjectId(10),

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Remove or filter disallowed URLs before enqueuing them.
2. If permitted by the site owner and your legal/policy constraints, change the crawler user agent to one robots.txt allows.
3. If robots.txt compliance is unwanted for your use case, disable the robots.txt feature in crawler options.
4. Handle skipped requests via the skipped-request handling hook so they are processed instead of thrown into your error handlers.

Example fix

// before
await crawler.addRequests([{ url: 'https://example.com/private/page' }]);
// after (filter against robots rules first)
const allowed = await robotsFile.isAllowed('https://example.com/private/page', crawlerUserAgent);
if (allowed) await crawler.addRequests([{ url: 'https://example.com/private/page' }]);

// Check robots.txt before enqueuing
const { RobotsTxtFile } = await import('crawlee');
const robots = await RobotsTxtFile.find('https://example.com');
const urls = ['https://example.com/a', 'https://example.com/private'];
const allowed = urls.filter((u) => robots.isAllowed(u, userAgent));
await crawler.addRequests(allowed.map((url) => ({ url })));

function isAllowedByRobots(robots, url, userAgent) { return robots.isAllowed(url, userAgent) === true; }

try {
  await crawler.run(requests);
} catch (err) {
  if (err.name === 'ContextPipelineInterruptedError' && err.message.includes('robots.txt')) {
    // record request.url as disallowed; do not retry
  } else throw err;
}

When it happens

Trigger: Enqueuing a URL whose robots.txt (for the crawler's user agent) disallows crawling it, while the robots.txt feature is enabled; the URL is checked before fetch in the crawl pipeline.

Common situations: Crawling sites that block bots via robots.txt; using a user agent that is disallowed while others are allowed; crawling paths added via addRequests that were never vetted against robots rules.

Related errors

• Skipping request ${request.id} (starting url: ${request.url}
• Failed to load robots.txt from ${url}: HTTP ${response.statu

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/2330de64785f2712. Report an issue: GitHub.