{"record":{"id":"f8c3581875053777","repo":"apify/crawlee","slug":"the-state-object-is-not-consistent-with-requestlis","errorCode":null,"errorMessage":"The state object is not consistent with RequestList, too few requests loaded.","messagePattern":"The state object is not consistent with RequestList, too few requests loaded\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/storages/request_list.ts","lineNumber":507,"sourceCode":"    private async persistRequests(): Promise<void> {\n        const serializedRequests = await serializeArray(this.requests);\n        this.#store ??= await KeyValueStore.open();\n        await this.#store.setValue(this.#persistRequestsKey!, serializedRequests, { contentType: CONTENT_TYPE_BINARY });\n        this.areRequestsPersisted = true;\n    }\n\n    /**\n     * Restores RequestList state from a state object.\n     */\n    private restoreState(state?: RequestListState): void {\n        // If there's no state it means we've not persisted any (yet).\n        if (!state) return;\n        // Restore previous state.\n        if (typeof state.nextIndex !== 'number' || state.nextIndex < 0) {\n            throw new Error('The state object is invalid: nextIndex must be a non-negative number.');\n        }\n        if (state.nextIndex > this.requests.length) {\n            throw new Error('The state object is not consistent with RequestList, too few requests loaded.');\n        }\n        if (\n            state.nextIndex < this.requests.length &&\n            this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey\n        ) {\n            throw new Error(\n                'The state object is not consistent with RequestList the order of URLs seems to have changed.',\n            );\n        }\n\n        const deleteFromInProgress: string[] = [];\n        state.inProgress.forEach((uniqueKey) => {\n            const index = this.#uniqueKeyToIndex[uniqueKey];\n            if (typeof index !== 'number') {\n                throw new Error(\n                    'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',\n                );\n            }","sourceCodeStart":489,"sourceCodeEnd":525,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/storages/request_list.ts#L489-L525","documentation":"Beyond being well-formed, the restored nextIndex must not exceed the number of loaded requests — otherwise the list cannot resume where it left off. This error means the persisted state references more requests than the current sources produced (state.nextIndex > this.requests.length).","triggerScenarios":"Re-running with fewer/smaller sources than the previous run (sources array trimmed, sourcesFunction returning fewer URLs), while a state persisted from the larger run is restored; switching persistStateKey between lists so state belongs to a different request set.","commonSituations":"Editing the sources list between deployments without clearing persisted state; a dynamic sourcesFunction returning fewer items due to an upstream API change; reusing the same persistStateKey for a different RequestList.","solutions":["Align sources so the list contains at least nextIndex requests (restore the removed URLs)","Delete the persisted state record to start fresh, or use a new persistStateKey for the changed list","Verify sourcesFunction returns the same/complete set of URLs as the run that persisted the state"],"exampleFix":"// before\n// state { nextIndex: 500 } but sources now produce 300 requests\n// after\nconst store = await KeyValueStore.open();\nawait store.setValue('CRAWLEE_list-state', null); // reset state to match the new sources\n// or restore the full sources array so length >= nextIndex","handlingStrategy":"try-catch","validationCode":"const stored = await store.getValue('CRAWLEE_list-state');\nif (stored && typeof stored.nextIndex === 'number' && sources && stored.nextIndex > sources.length) {\n  await store.setValue('CRAWLEE_list-state', null); // state belongs to a larger source set — reset\n}","typeGuard":null,"tryCatchPattern":"try {\n  const list = await RequestList.open('list', { sources, persistStateKey: 'state' });\n} catch (e) {\n  if (e.message.includes('too few requests loaded')) {\n    const store = await KeyValueStore.open();\n    await store.setValue('CRAWLEE_state', null);\n    return RequestList.open('list', { sources, persistStateKey: 'state' });\n  }\n  throw e;\n}","preventionTips":["Keep the sources list stable across runs while state is persisted","Use a distinct persistStateKey whenever you materially change sources","Clear persisted state when intentionally shrinking the source list"],"tags":["request-list","state","consistency","corrupted-state"],"backgroundTag":"invalid-persisted-state","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}