{"record":{"id":"375f8143920fa58f","repo":"apify/crawlee","slug":"the-state-object-is-not-consistent-with-requestlis-375f81","errorCode":null,"errorMessage":"The state object is not consistent with RequestList the order of URLs seems to have changed.","messagePattern":"The state object is not consistent with RequestList the order of URLs seems to have changed\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/storages/request_list.ts","lineNumber":513,"sourceCode":"\n    /**\n     * Restores RequestList state from a state object.\n     */\n    private restoreState(state?: RequestListState): void {\n        // If there's no state it means we've not persisted any (yet).\n        if (!state) return;\n        // Restore previous state.\n        if (typeof state.nextIndex !== 'number' || state.nextIndex < 0) {\n            throw new Error('The state object is invalid: nextIndex must be a non-negative number.');\n        }\n        if (state.nextIndex > this.requests.length) {\n            throw new Error('The state object is not consistent with RequestList, too few requests loaded.');\n        }\n        if (\n            state.nextIndex < this.requests.length &&\n            this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey\n        ) {\n            throw new Error(\n                'The state object is not consistent with RequestList the order of URLs seems to have changed.',\n            );\n        }\n\n        const deleteFromInProgress: string[] = [];\n        state.inProgress.forEach((uniqueKey) => {\n            const index = this.#uniqueKeyToIndex[uniqueKey];\n            if (typeof index !== 'number') {\n                throw new Error(\n                    'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',\n                );\n            }\n            if (index >= state.nextIndex) {\n                deleteFromInProgress.push(uniqueKey);\n            }\n        });\n\n        this.#nextIndex = state.nextIndex;","sourceCodeStart":495,"sourceCodeEnd":531,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/storages/request_list.ts#L495-L531","documentation":"RequestList.restoreState() validates that a previously persisted state (nextIndex + nextUniqueKey) still matches the current request ordering. If the request at state.nextIndex has a different uniqueKey than state.nextUniqueKey, the URL order changed and restoring would silently skip or duplicate work, so the library throws. This typically happens when the source list changed between runs while using persisted state.","triggerScenarios":"Calling await requestList.initialize() with persistStateKey/persistRequestsKey where the underlying requests source (list of URLs, requestsFromUrl resource, or order of addRequests calls) was modified, reordered, or shortened since the state was persisted; passing a state object obtained from a different RequestList instance.","commonSituations":"Reordering entries in a remote URL list file consumed via requestsFromUrl; editing a hardcoded URL array between deploys; restoring state saved by a different crawler version or different list; concurrent runs sharing a persisted state key.","solutions":["Make the request source deterministic and identical across runs (same URLs, same order) before re-initializing","Delete the persisted state (and optionally persisted requests) under the configured persistStateKey so a fresh state is created","Use keepUniqueKeys/keepDuplicateRequests semantics carefully or compute stable uniqueKeys so ordering changes are intentional","If order changes are expected, stop using state persistence for that RequestList"],"exampleFix":"// before (order changed between runs, stale state)\nconst list = await RequestList.open('my-list', urlsReorderedThisRun);\n// after: use a new state key or clear old persisted state when the list changes\nconst list = await RequestList.open(orderChanged ? 'my-list-v2' : 'my-list', urls);","handlingStrategy":"validation","validationCode":"function isStateConsistent(list, state) {\n  return state.nextIndex < 0 || list.requests?.[state.nextIndex]?.uniqueKey === state.nextUniqueKey;\n}","typeGuard":"function hasValidStateShape(state) {\n  return typeof state?.nextIndex === 'number' && typeof state?.nextUniqueKey === 'string' && Array.isArray(state?.inProgress);\n}","tryCatchPattern":"try {\n  await requestList.initialize();\n} catch (err) {\n  if (err.message.includes('not consistent with RequestList')) {\n    // clear persisted state and retry once with fresh state\n    await purgeCachedStorage({ name: 'default' });\n    await requestList.initialize();\n  } else throw err;\n}","preventionTips":["Treat the request source as immutable once a persisted state exists","Version the persistStateKey whenever the list content changes","Never share a state key between different lists or runs"],"tags":["state","persistence","request-list"],"backgroundTag":"state-inconsistent-with-source","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}