{"record":{"id":"2330de64785f2712","repo":"apify/crawlee","slug":"skipping-request-request-url-as-disallowed-by-r","errorCode":null,"errorMessage":"Skipping request ${request.url} as disallowed by robots.txt","messagePattern":"Skipping request (.+?) as disallowed by robots\\.txt","errorType":"exception","errorClass":"ContextPipelineInterruptedError","httpStatus":null,"severity":"warning","filePath":"packages/basic-crawler/src/internals/basic-crawler.ts","lineNumber":1463,"sourceCode":"            .compose({ action: this.checkRobotsTxt.bind(this) })\n            .compose({ action: (context) => this.createBaseContext(context) })\n            .compose({ action: this.resolveSession.bind(this) })\n            .compose({ action: this.createContextHelpers.bind(this) });\n    }\n\n    private async checkRobotsTxt({ request }: { request: Request }) {\n        if (!(await this.isAllowedBasedOnRobotsTxtFile(request.url))) {\n            this.log.warning(\n                `Skipping request ${request.url} (${request.id}) because it is disallowed based on robots.txt`,\n            );\n            request.state = RequestState.SKIPPED;\n            request.noRetry = true;\n            await this.#handleSkippedRequest({\n                request,\n                reason: 'robotsTxt',\n            });\n\n            throw new ContextPipelineInterruptedError(`Skipping request ${request.url} as disallowed by robots.txt`);\n        }\n\n        return {};\n    }\n\n    /**\n     * Builds the subclass-specific context pipeline that transforms a `CrawlingContext` into the crawler's target context type.\n     * Subclasses should override this to add their own pipeline stages.\n     */\n    protected buildContextPipeline(): ContextPipeline<CrawlingContext, CrawlingContext> {\n        return ContextPipeline.create<CrawlingContext>();\n    }\n\n    private createBaseContext(context: PendingCrawlingContext) {\n        const deferredCleanup: (() => Promise<unknown>)[] = [];\n\n        return {\n            id: cryptoRandomObjectId(10),","sourceCodeStart":1445,"sourceCodeEnd":1481,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/basic-crawler/src/internals/basic-crawler.ts#L1445-L1481","documentation":"When robots.txt analysis (robotsTxtFile feature) marks a URL as disallowed for the crawler's user agent, the crawler refuses to fetch it, records it as a skipped request, and interrupts the context pipeline with ContextPipelineInterruptedError. The request is marked noRetry so it is not retried.","triggerScenarios":"Enqueuing a URL whose robots.txt (for the crawler's user agent) disallows crawling it, while the robots.txt feature is enabled; the URL is checked before fetch in the crawl pipeline.","commonSituations":"Crawling sites that block bots via robots.txt; using a user agent that is disallowed while others are allowed; crawling paths added via addRequests that were never vetted against robots rules.","solutions":["Remove or filter disallowed URLs before enqueuing them.","If permitted by the site owner and your legal/policy constraints, change the crawler user agent to one robots.txt allows.","If robots.txt compliance is unwanted for your use case, disable the robots.txt feature in crawler options.","Handle skipped requests via the skipped-request handling hook so they are processed instead of thrown into your error handlers."],"exampleFix":"// before\nawait crawler.addRequests([{ url: 'https://example.com/private/page' }]);\n// after (filter against robots rules first)\nconst allowed = await robotsFile.isAllowed('https://example.com/private/page', crawlerUserAgent);\nif (allowed) await crawler.addRequests([{ url: 'https://example.com/private/page' }]);","handlingStrategy":"validation","validationCode":"// Check robots.txt before enqueuing\nconst { RobotsTxtFile } = await import('crawlee');\nconst robots = await RobotsTxtFile.find('https://example.com');\nconst urls = ['https://example.com/a', 'https://example.com/private'];\nconst allowed = urls.filter((u) => robots.isAllowed(u, userAgent));\nawait crawler.addRequests(allowed.map((url) => ({ url })));","typeGuard":"function isAllowedByRobots(robots, url, userAgent) { return robots.isAllowed(url, userAgent) === true; }","tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  if (err.name === 'ContextPipelineInterruptedError' && err.message.includes('robots.txt')) {\n    // record request.url as disallowed; do not retry\n  } else throw err;\n}","preventionTips":["Filter candidate URLs against robots.txt before enqueuing.","Use a user agent that robots.txt permits for your crawler.","Route disallowed URLs to the skipped-request handler instead of letting the pipeline interrupt.","Document which target sites allow crawling and maintain allowlists."],"tags":["robots-txt","skipped-request","crawling-policy"],"backgroundTag":"robots-txt-disallowed","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}