apify/crawlee · error · PersistentRateLimitError

Giving up: ${state.reason}

Error message

Giving up: ${state.reason}

What it means

When the request manager reports its status as 'stalled' — it can make no progress (e.g. every source exhausted or persistently rate-limited) — the crawler throws PersistentRateLimitError to end the run, but only when `keepAlive` is false. With keepAlive enabled the crawler intentionally outlives the stall and keeps waiting.

Source

Thrown at packages/basic-crawler/src/internals/basic-crawler.ts:1376

                        );
                        return true;
                    }

                    if (this.#unexpectedStop) {
                        this.log.info(
                            'The crawler has finished all the remaining ongoing requests and will shut down now.',
                        );
                        return true;
                    }

                    // `maybeFinish()` calls this only once nothing is in flight (`autoscaled_pool.ts`) - the point
                    // where a crawl that cannot progress becomes distinguishable from one that is merely waiting,
                    // and the only place where throwing does not abandon requests mid-processing.
                    const state = await this.requestManager?.checkReadiness();

                    // Under `keepAlive`, outliving a domain that will not let us through is the whole point.
                    if (state?.status === 'stalled' && !keepAlive) {
                        throw new PersistentRateLimitError(`Giving up: ${state.reason}`);
                    }

                    const isFinished = isFinishedFunction
                        ? await isFinishedFunction()
                        : state === undefined || state.status === 'finished';

                    if (isFinished) {
                        const reason = isFinishedFunction
                            ? "Crawler's custom isFinishedFunction() returned true, the crawler will shut down."
                            : 'All requests from the queue have been processed, the crawler will shut down.';
                        this.log.info(reason);
                    }

                    return isFinished;
                },
                log: this.log,
            };

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Inspect `state.reason` in the message to see why the manager is stalled (rate limits, empty sources) and fix that underlying cause.
2. Respect robots.txt / reduce request rate so the domain stops blocking the crawler.
3. If the crawl should survive stalls, pass `keepAlive: true` to the crawler/run options.
4. Ensure the request source (list/queue) is properly marked finished or not-done so status is 'finished' rather than 'stalled'.

Example fix

// before
const stats = await crawler.run(requests);
// after (keep the crawler alive through stalls)
const stats = await crawler.run(requests, { keepAlive: true });

import { PersistentRateLimitError } from '@crawlee/basic';
try {
  await crawler.run(requests);
} catch (err) {
  if (err instanceof PersistentRateLimitError) {
    // log err.message's reason; schedule a later rerun or enable keepAlive
  } else throw err;
}

When it happens

Trigger: `crawler.run()` without `keepAlive: true` while `requestManager.checkReadiness()` returns `{ status: 'stalled', reason }` — e.g. a request queue permanently blocked by rate limiting or an exhausted request list.

Common situations: Heavy rate limiting (HTTP 429s) stalling the per-domain queues; a request list that is exhausted but not marked finished; waiting on sources that never become ready.

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/e2d3553037e8ca34. Report an issue: GitHub.