{"record":{"id":"e2d3553037e8ca34","repo":"apify/crawlee","slug":"giving-up-state-reason","errorCode":null,"errorMessage":"Giving up: ${state.reason}","messagePattern":"Giving up: (.+?)","errorType":"exception","errorClass":"PersistentRateLimitError","httpStatus":null,"severity":"error","filePath":"packages/basic-crawler/src/internals/basic-crawler.ts","lineNumber":1376,"sourceCode":"                        );\n                        return true;\n                    }\n\n                    if (this.#unexpectedStop) {\n                        this.log.info(\n                            'The crawler has finished all the remaining ongoing requests and will shut down now.',\n                        );\n                        return true;\n                    }\n\n                    // `maybeFinish()` calls this only once nothing is in flight (`autoscaled_pool.ts`) - the point\n                    // where a crawl that cannot progress becomes distinguishable from one that is merely waiting,\n                    // and the only place where throwing does not abandon requests mid-processing.\n                    const state = await this.requestManager?.checkReadiness();\n\n                    // Under `keepAlive`, outliving a domain that will not let us through is the whole point.\n                    if (state?.status === 'stalled' && !keepAlive) {\n                        throw new PersistentRateLimitError(`Giving up: ${state.reason}`);\n                    }\n\n                    const isFinished = isFinishedFunction\n                        ? await isFinishedFunction()\n                        : state === undefined || state.status === 'finished';\n\n                    if (isFinished) {\n                        const reason = isFinishedFunction\n                            ? \"Crawler's custom isFinishedFunction() returned true, the crawler will shut down.\"\n                            : 'All requests from the queue have been processed, the crawler will shut down.';\n                        this.log.info(reason);\n                    }\n\n                    return isFinished;\n                },\n                log: this.log,\n            };\n","sourceCodeStart":1358,"sourceCodeEnd":1394,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/basic-crawler/src/internals/basic-crawler.ts#L1358-L1394","documentation":"When the request manager reports its status as 'stalled' — it can make no progress (e.g. every source exhausted or persistently rate-limited) — the crawler throws PersistentRateLimitError to end the run, but only when `keepAlive` is false. With keepAlive enabled the crawler intentionally outlives the stall and keeps waiting.","triggerScenarios":"`crawler.run()` without `keepAlive: true` while `requestManager.checkReadiness()` returns `{ status: 'stalled', reason }` — e.g. a request queue permanently blocked by rate limiting or an exhausted request list.","commonSituations":"Heavy rate limiting (HTTP 429s) stalling the per-domain queues; a request list that is exhausted but not marked finished; waiting on sources that never become ready.","solutions":["Inspect `state.reason` in the message to see why the manager is stalled (rate limits, empty sources) and fix that underlying cause.","Respect robots.txt / reduce request rate so the domain stops blocking the crawler.","If the crawl should survive stalls, pass `keepAlive: true` to the crawler/run options.","Ensure the request source (list/queue) is properly marked finished or not-done so status is 'finished' rather than 'stalled'."],"exampleFix":"// before\nconst stats = await crawler.run(requests);\n// after (keep the crawler alive through stalls)\nconst stats = await crawler.run(requests, { keepAlive: true });","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"import { PersistentRateLimitError } from '@crawlee/basic';\ntry {\n  await crawler.run(requests);\n} catch (err) {\n  if (err instanceof PersistentRateLimitError) {\n    // log err.message's reason; schedule a later rerun or enable keepAlive\n  } else throw err;\n}","preventionTips":["Set request rates within robots/rate-limit budgets to avoid persistent stalls.","Enable keepAlive for long-lived crawlers expected to wait out stalls.","Monitor request manager readiness/status during runs.","Always read the `reason` in the error message to fix the stall cause before rerunning."],"tags":["rate-limit","stalled","keep-alive"],"backgroundTag":"persistent-rate-limit","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}