{"record":{"id":"bc6cf1fea73f670c","repo":"apify/crawlee","slug":"the-state-object-is-not-consistent-with-requestlis-bc6cf1","errorCode":null,"errorMessage":"The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.","messagePattern":"The state object is not consistent with RequestList\\. Unknown uniqueKey is present in the state\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/storages/request_list.ts","lineNumber":522,"sourceCode":"            throw new Error('The state object is invalid: nextIndex must be a non-negative number.');\n        }\n        if (state.nextIndex > this.requests.length) {\n            throw new Error('The state object is not consistent with RequestList, too few requests loaded.');\n        }\n        if (\n            state.nextIndex < this.requests.length &&\n            this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey\n        ) {\n            throw new Error(\n                'The state object is not consistent with RequestList the order of URLs seems to have changed.',\n            );\n        }\n\n        const deleteFromInProgress: string[] = [];\n        state.inProgress.forEach((uniqueKey) => {\n            const index = this.#uniqueKeyToIndex[uniqueKey];\n            if (typeof index !== 'number') {\n                throw new Error(\n                    'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',\n                );\n            }\n            if (index >= state.nextIndex) {\n                deleteFromInProgress.push(uniqueKey);\n            }\n        });\n\n        this.#nextIndex = state.nextIndex;\n        this.inProgress = new Set(state.inProgress);\n\n        // WORKAROUND:\n        // It happened to some users that state object contained something like:\n        // {\n        //   \"nextIndex\": 11308,\n        //   \"nextUniqueKey\": \"https://www.anychart.com\",\n        //   \"inProgress\": {\n        //      \"https://www.ams360.com\": true,","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/storages/request_list.ts#L504-L540","documentation":"restoreState() looks up each uniqueKey recorded in state.inProgress in the RequestList's uniqueKey-to-index map. If a key is not found, the state references a request that no longer exists in this list, so the state is invalid and the library throws rather than resuming against an unknown request.","triggerScenarios":"Calling initialize() with persisted state whose inProgress set contains uniqueKeys absent from the current request list — e.g. requests were removed, the source list changed, the state came from a different RequestList, or uniqueKey computation (keepDuplicateRequests / deduplication) changed.","commonSituations":"Removing URLs from a list while crawler runs persisted in-progress requests; swapping persistRequestsKey sources; migrating between library versions that changed uniqueKey computation; copying state files between projects.","solutions":["Clear the persisted state so initialize() builds a fresh one","Ensure the request list contains exactly the same requests (same uniqueKeys) as when the state was saved","Verify keepDuplicateRequests and request construction are unchanged, since they affect uniqueKey generation","Do not share a persistStateKey between different RequestList instances"],"exampleFix":"// before: reusing state key after changing the URL list\nawait RequestList.open('crawl', fewerUrls); // throws on initialize\n// after: new key per list content version\nawait RequestList.open('crawl-v3', fewerUrls);","handlingStrategy":"validation","validationCode":"const known = new Set(listRequests.map(r => r.uniqueKey));\nif (state.inProgress.some(k => !known.has(k))) throw new Error('State references unknown requests');","typeGuard":"function stateKeysKnown(state, list) {\n  return state.inProgress.every(k => typeof k === 'string' && list.hasUniqueKey?.(k) !== false);\n}","tryCatchPattern":"try {\n  await requestList.initialize();\n} catch (err) {\n  if (err.message.includes('Unknown uniqueKey')) {\n    // discard incompatible state, start fresh\n    await requestList.initialize({ force: true }); // or purge state storage first\n  } else throw err;\n}","preventionTips":["Keep the request set stable across runs that share persisted state","Don't copy state files between RequestList instances or projects","Verify uniqueKey computation settings (keepDuplicateRequests) don't change between versions"],"tags":["state","persistence","request-list"],"backgroundTag":"state-inconsistent-with-source","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}