apify/crawlee · error · Error

Cannot honour a crawl-delay floor covering every domain: thi

Error message

Cannot honour a crawl-delay floor covering every domain: this manager only paces the domains it was given (${Array.from(this.#listedDomains).join(', ')}), so everything else would run unpaced. Set `domains: 'all'` to pace whatever the crawl encounters, or declare the floor on this manager yourself via `minCrawlDelaySecs`.

What it means

When a server sends a crawl-delay (robots.txt) pacing signal scoped to all domains, the manager refuses to apply it unless it actually throttles every domain (`domains: 'all'`). With an explicit domain list, other domains would run unpaced, so it throws instead of silently under-pacing.

Source

Thrown at packages/core/src/storages/throttling_request_manager.ts:634

    /**
     * The runtime form of {@apilink ThrottlingRequestManagerOptions.minCrawlDelaySecs|`minCrawlDelaySecs`}: raises
     * the floor under every throttled domain's crawl delay.
     *
     * @returns `false` if this manager paces nothing at all, so there is no floor to raise.
     * @throws If it paces some domains but not all - holding back only those would leave the rest of what the
     *  floor covers unpaced.
     */
    #recordEverywhereFloor(intervalMs: number, scope: PacingScope): boolean {
        // Before the scope check: a manager that paces nothing has no grouping worth objecting about, and
        // `false` lets the caller pace it from outside.
        if (!this.#throttlingEnabled) {
            return false;
        }

        this.#assertScopeHonourable(scope);

        if (!this.#throttlesEveryDomain) {
            throw new Error(
                `Cannot honour a crawl-delay floor covering every domain: this manager only paces the domains ` +
                    `it was given (${Array.from(this.#listedDomains).join(', ')}), so everything else would run ` +
                    `unpaced. Set \`domains: 'all'\` to pace whatever the crawl encounters, or declare the floor ` +
                    `on this manager yourself via \`minCrawlDelaySecs\`.`,
            );
        }

        this.#minCrawlDelayMs = Math.max(this.#minCrawlDelayMs, intervalMs);
        this.log.debug(`Crawl-delay floor for every domain set to ${(this.#minCrawlDelayMs / 1000).toFixed(1)}s`);

        return true;
    }

    /**
     * Throws unless a signal scoped to `scope` can be honoured as declared or wider.
     *
     * Holding a domain back means holding its queue back, so
     * {@apilink ThrottlingRequestManagerOptions.throttleBy|`throttleBy`} is the finest granularity this manager

View on GitHub (pinned to dbe57fb09c)

Solutions

  1. Construct the manager with `domains: 'all'` so it can pace any encountered domain
  2. Set `minCrawlDelaySecs` on the manager yourself to declare the floor explicitly
  3. Narrow the signal scope so it targets only the domains the manager lists
  4. Add the relevant domain(s) to `domains` if partial coverage is actually intended (then scope the signal accordingly)

Example fix

// before
const mgr = new ThrottlingRequestManager({ domains: ['example.com'] });
mgr.recordPacingSignal({ scope: 'everywhere', minDelaySecs: 5 }); // throws
// after
const mgr = new ThrottlingRequestManager({ domains: 'all', minCrawlDelaySecs: 5 });
Defensive patterns

Strategy: try-catch

Validate before calling

const canHonourEverywhere =
  managerConfig.domains === 'all' || managerConfig.minCrawlDelaySecs != null;
if (signal.scope === 'everywhere' && !canHonourEverywhere) {
  // skip signal or reconfigure manager before calling recordPacingSignal
}

Type guard

null

Try / catch

try {
  manager.recordPacingSignal(signal);
} catch (err) {
  if (/Cannot honour a crawl-delay floor/.test(String(err))) {
    logger.warning('Crawl-delay floor ignored: manager scope too narrow');
  } else throw err;
}

Prevention

When it happens

Trigger: Calling recordPacingSignal with an everywhere/registrable-wide crawl-delay floor while the manager was constructed with an explicit `domains` array (not 'all').

Common situations: Robots.txt of some site declares a global crawl-delay and the crawler relays it to a manager configured for a fixed domain list; developers assume the manager applies floors globally by default.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/885adf5246faf3e8. Report an issue: GitHub.