{"record":{"id":"885adf5246faf3e8","repo":"apify/crawlee","slug":"cannot-honour-a-crawl-delay-floor-covering-every-d","errorCode":null,"errorMessage":"Cannot honour a crawl-delay floor covering every domain: this manager only paces the domains it was given (${Array.from(this.#listedDomains).join(', ')}), so everything else would run unpaced. Set `domains: 'all'` to pace whatever the crawl encounters, or declare the floor on this manager yourself via `minCrawlDelaySecs`.","messagePattern":"Cannot honour a crawl-delay floor covering every domain: this manager only paces the domains it was given \\((.+?)\\), so everything else would run unpaced\\. Set `domains: 'all'` to pace whatever the crawl encounters, or declare the floor on this manager yourself via `minCrawlDelaySecs`\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/storages/throttling_request_manager.ts","lineNumber":634,"sourceCode":"    /**\n     * The runtime form of {@apilink ThrottlingRequestManagerOptions.minCrawlDelaySecs|`minCrawlDelaySecs`}: raises\n     * the floor under every throttled domain's crawl delay.\n     *\n     * @returns `false` if this manager paces nothing at all, so there is no floor to raise.\n     * @throws If it paces some domains but not all - holding back only those would leave the rest of what the\n     *  floor covers unpaced.\n     */\n    #recordEverywhereFloor(intervalMs: number, scope: PacingScope): boolean {\n        // Before the scope check: a manager that paces nothing has no grouping worth objecting about, and\n        // `false` lets the caller pace it from outside.\n        if (!this.#throttlingEnabled) {\n            return false;\n        }\n\n        this.#assertScopeHonourable(scope);\n\n        if (!this.#throttlesEveryDomain) {\n            throw new Error(\n                `Cannot honour a crawl-delay floor covering every domain: this manager only paces the domains ` +\n                    `it was given (${Array.from(this.#listedDomains).join(', ')}), so everything else would run ` +\n                    `unpaced. Set \\`domains: 'all'\\` to pace whatever the crawl encounters, or declare the floor ` +\n                    `on this manager yourself via \\`minCrawlDelaySecs\\`.`,\n            );\n        }\n\n        this.#minCrawlDelayMs = Math.max(this.#minCrawlDelayMs, intervalMs);\n        this.log.debug(`Crawl-delay floor for every domain set to ${(this.#minCrawlDelayMs / 1000).toFixed(1)}s`);\n\n        return true;\n    }\n\n    /**\n     * Throws unless a signal scoped to `scope` can be honoured as declared or wider.\n     *\n     * Holding a domain back means holding its queue back, so\n     * {@apilink ThrottlingRequestManagerOptions.throttleBy|`throttleBy`} is the finest granularity this manager","sourceCodeStart":616,"sourceCodeEnd":652,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/storages/throttling_request_manager.ts#L616-L652","documentation":"When a server sends a crawl-delay (robots.txt) pacing signal scoped to all domains, the manager refuses to apply it unless it actually throttles every domain (`domains: 'all'`). With an explicit domain list, other domains would run unpaced, so it throws instead of silently under-pacing.","triggerScenarios":"Calling recordPacingSignal with an everywhere/registrable-wide crawl-delay floor while the manager was constructed with an explicit `domains` array (not 'all').","commonSituations":"Robots.txt of some site declares a global crawl-delay and the crawler relays it to a manager configured for a fixed domain list; developers assume the manager applies floors globally by default.","solutions":["Construct the manager with `domains: 'all'` so it can pace any encountered domain","Set `minCrawlDelaySecs` on the manager yourself to declare the floor explicitly","Narrow the signal scope so it targets only the domains the manager lists","Add the relevant domain(s) to `domains` if partial coverage is actually intended (then scope the signal accordingly)"],"exampleFix":"// before\nconst mgr = new ThrottlingRequestManager({ domains: ['example.com'] });\nmgr.recordPacingSignal({ scope: 'everywhere', minDelaySecs: 5 }); // throws\n// after\nconst mgr = new ThrottlingRequestManager({ domains: 'all', minCrawlDelaySecs: 5 });","handlingStrategy":"try-catch","validationCode":"const canHonourEverywhere =\n  managerConfig.domains === 'all' || managerConfig.minCrawlDelaySecs != null;\nif (signal.scope === 'everywhere' && !canHonourEverywhere) {\n  // skip signal or reconfigure manager before calling recordPacingSignal\n}","typeGuard":"null","tryCatchPattern":"try {\n  manager.recordPacingSignal(signal);\n} catch (err) {\n  if (/Cannot honour a crawl-delay floor/.test(String(err))) {\n    logger.warning('Crawl-delay floor ignored: manager scope too narrow');\n  } else throw err;\n}","preventionTips":["Use domains: 'all' when relaying robots.txt crawl-delays","Declare minCrawlDelaySecs explicitly if you need a global floor","Keep pacing-signal emitters aware of the manager's domains configuration"],"tags":["throttling","configuration","crawl-delay"],"backgroundTag":"pacing-scope-unhonourable","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}