apify/crawlee · error · Error

The state object is not consistent with RequestList. Unknown

Error message

The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.

What it means

restoreState() looks up each uniqueKey recorded in state.inProgress in the RequestList's uniqueKey-to-index map. If a key is not found, the state references a request that no longer exists in this list, so the state is invalid and the library throws rather than resuming against an unknown request.

Source

Thrown at packages/core/src/storages/request_list.ts:522

            throw new Error('The state object is invalid: nextIndex must be a non-negative number.');
        }
        if (state.nextIndex > this.requests.length) {
            throw new Error('The state object is not consistent with RequestList, too few requests loaded.');
        }
        if (
            state.nextIndex < this.requests.length &&
            this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey
        ) {
            throw new Error(
                'The state object is not consistent with RequestList the order of URLs seems to have changed.',
            );
        }

        const deleteFromInProgress: string[] = [];
        state.inProgress.forEach((uniqueKey) => {
            const index = this.#uniqueKeyToIndex[uniqueKey];
            if (typeof index !== 'number') {
                throw new Error(
                    'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',
                );
            }
            if (index >= state.nextIndex) {
                deleteFromInProgress.push(uniqueKey);
            }
        });

        this.#nextIndex = state.nextIndex;
        this.inProgress = new Set(state.inProgress);

        // WORKAROUND:
        // It happened to some users that state object contained something like:
        // {
        //   "nextIndex": 11308,
        //   "nextUniqueKey": "https://www.anychart.com",
        //   "inProgress": {
        //      "https://www.ams360.com": true,

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Clear the persisted state so initialize() builds a fresh one
2. Ensure the request list contains exactly the same requests (same uniqueKeys) as when the state was saved
3. Verify keepDuplicateRequests and request construction are unchanged, since they affect uniqueKey generation
4. Do not share a persistStateKey between different RequestList instances

Example fix

// before: reusing state key after changing the URL list
await RequestList.open('crawl', fewerUrls); // throws on initialize
// after: new key per list content version
await RequestList.open('crawl-v3', fewerUrls);

const known = new Set(listRequests.map(r => r.uniqueKey));
if (state.inProgress.some(k => !known.has(k))) throw new Error('State references unknown requests');

function stateKeysKnown(state, list) {
  return state.inProgress.every(k => typeof k === 'string' && list.hasUniqueKey?.(k) !== false);
}

try {
  await requestList.initialize();
} catch (err) {
  if (err.message.includes('Unknown uniqueKey')) {
    // discard incompatible state, start fresh
    await requestList.initialize({ force: true }); // or purge state storage first
  } else throw err;
}

When it happens

Trigger: Calling initialize() with persisted state whose inProgress set contains uniqueKeys absent from the current request list — e.g. requests were removed, the source list changed, the state came from a different RequestList, or uniqueKey computation (keepDuplicateRequests / deduplication) changed.

Common situations: Removing URLs from a list while crawler runs persisted in-progress requests; swapping persistRequestsKey sources; migrating between library versions that changed uniqueKey computation; copying state files between projects.

Related errors

• The state object is not consistent with RequestList the orde
• Cannot clear the state persisted under key '${this.#persistS
• Cannot persist state. options.persistStateKey is not set.
• The state object is invalid: nextIndex must be a non-negativ
• The state object is not consistent with RequestList, too few

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/bc6cf1fea73f670c. Report an issue: GitHub.