apify/crawlee · error · Error

Cannot decide what to purge before running again: `sameDomai

Error message

Cannot decide what to purge before running again: `sameDomainDelaySecs` paces the request manager you supplied, so the per-domain queues that have to be emptied are the crawler's while the manager underneath them is yours. Say which you want: `run(requests, { purgeRequestQueue: true })` empties both, `false` empties neither.

What it means

When the crawler paces domains via `sameDomainDelaySecs`, that delay lives in the request manager. If the user supplied their own requestManager, purging on re-run becomes ambiguous: the crawler-owned per-domain queues vs. the caller's manager. `purgeRequestQueue` left undefined cannot decide, so run() throws and demands an explicit choice.

Source

Thrown at packages/basic-crawler/src/internals/basic-crawler.ts:1715

     */
    async run(requests?: TypedRequestsLike<Routes>, options?: CrawlerRunOptions): Promise<FinalStatistics> {
        if (this.running) {
            throw new Error(
                'This crawler instance is already running, you can add more requests to it via `crawler.addRequests()`.',
            );
        }

        const { purgeRequestQueue, ...addRequestsOptions } = options ?? {};

        if (this.hasFinishedBefore) {
            // When executing the run method for the second time explicitly,
            // we need to purge the RQ to allow processing the same requests again — this is important so users can
            // pass in failed requests back to the `crawler.run()`, otherwise they would be considered as handled and
            // ignored — as a failed request is still handled.
            // `purgeRequestQueue` unset purges only storage the crawler opened itself (see `#purgeableExtent`);
            // `true` also purges a caller-supplied manager, `false` purges nothing.
            if (purgeRequestQueue === undefined && this.#purgeableExtent === 'ambiguous') {
                throw new Error(
                    'Cannot decide what to purge before running again: `sameDomainDelaySecs` paces the request ' +
                        'manager you supplied, so the per-domain queues that have to be emptied are the ' +
                        "crawler's while the manager underneath them is yours. Say which you want: " +
                        '`run(requests, { purgeRequestQueue: true })` empties both, `false` empties neither.',
                );
            }

            if (purgeRequestQueue !== false && (this.#purgeableExtent === 'all' || purgeRequestQueue === true)) {
                // One call from the outside in reaches everything the manager wraps, a pacer's per-domain queues
                // included.
                await this.requestManager?.purge?.();
            }

            // A supplied statistics instance keeps whatever state it was handed - only wipe a default we built.
            await this.#statisticsDep.ifOwned(async (stats) => {
                stats.reset();
                await stats.resetStore();
            });

View on GitHub (pinned to dbe57fb09c)

Solutions

Pass `purgeRequestQueue: true` to run() to purge both the crawler's queues and the supplied manager (full re-run).
Pass `purgeRequestQueue: false` to purge nothing if re-processing queued requests is unwanted.
On the first run of a fresh crawler, avoid the ambiguity by constructing with defaults instead of a supplied requestManager.

Example fix

// before
await crawler.run(failedRequests);
// after
await crawler.run(failedRequests, { purgeRequestQueue: true });

Defensive patterns

Strategy: validation

Validate before calling

const opts = { purgeRequestQueue: true };
if (crawlerUsesSuppliedManagerWithSameDomainDelay) {
  if (opts.purgeRequestQueue === undefined) {
    throw new Error('Specify purgeRequestQueue true/false for this re-run.');
  }
}
await crawler.run(requests, opts);

Type guard

function hasExplicitPurge(o) { return typeof o.purgeRequestQueue === 'boolean'; }

Try / catch

try {
  await crawler.run(requests);
} catch (err) {
  if (err.message.startsWith('Cannot decide what to purge')) {
    await crawler.run(requests, { purgeRequestQueue: true });
  } else throw err;
}

Prevention

Always pass an explicit purgeRequestQueue boolean when re-running a crawler that uses a supplied requestManager with sameDomainDelaySecs.
Use purgeRequestQueue: true when re-enqueuing failed requests so they are not treated as already handled.
Avoid mixing a custom requestManager with sameDomainDelaySecs unless you understand the purge implications.
Wrap repeated crawler.run() invocations in a helper that fixes purge semantics in one place.

When it happens

Trigger: Calling `crawler.run(requests)` a second time (re-run) on a crawler configured with `sameDomainDelaySecs` and a user-supplied `requestManager`, without passing an explicit `purgeRequestQueue` value.

Common situations: Re-running failed requests back through the same crawler instance; retry scripts that call run() repeatedly; switching to a custom requestManager while relying on the default purge behavior.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/8a7f13e9d45393de. Report an issue: GitHub.