apify/crawlee · error · Error

The state object is not consistent with RequestList the orde

Error message

The state object is not consistent with RequestList the order of URLs seems to have changed.

What it means

RequestList.restoreState() validates that a previously persisted state (nextIndex + nextUniqueKey) still matches the current request ordering. If the request at state.nextIndex has a different uniqueKey than state.nextUniqueKey, the URL order changed and restoring would silently skip or duplicate work, so the library throws. This typically happens when the source list changed between runs while using persisted state.

Source

Thrown at packages/core/src/storages/request_list.ts:513

    /**
     * Restores RequestList state from a state object.
     */
    private restoreState(state?: RequestListState): void {
        // If there's no state it means we've not persisted any (yet).
        if (!state) return;
        // Restore previous state.
        if (typeof state.nextIndex !== 'number' || state.nextIndex < 0) {
            throw new Error('The state object is invalid: nextIndex must be a non-negative number.');
        }
        if (state.nextIndex > this.requests.length) {
            throw new Error('The state object is not consistent with RequestList, too few requests loaded.');
        }
        if (
            state.nextIndex < this.requests.length &&
            this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey
        ) {
            throw new Error(
                'The state object is not consistent with RequestList the order of URLs seems to have changed.',
            );
        }

        const deleteFromInProgress: string[] = [];
        state.inProgress.forEach((uniqueKey) => {
            const index = this.#uniqueKeyToIndex[uniqueKey];
            if (typeof index !== 'number') {
                throw new Error(
                    'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',
                );
            }
            if (index >= state.nextIndex) {
                deleteFromInProgress.push(uniqueKey);
            }
        });

        this.#nextIndex = state.nextIndex;

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Make the request source deterministic and identical across runs (same URLs, same order) before re-initializing
2. Delete the persisted state (and optionally persisted requests) under the configured persistStateKey so a fresh state is created
3. Use keepUniqueKeys/keepDuplicateRequests semantics carefully or compute stable uniqueKeys so ordering changes are intentional
4. If order changes are expected, stop using state persistence for that RequestList

Example fix

// before (order changed between runs, stale state)
const list = await RequestList.open('my-list', urlsReorderedThisRun);
// after: use a new state key or clear old persisted state when the list changes
const list = await RequestList.open(orderChanged ? 'my-list-v2' : 'my-list', urls);

function isStateConsistent(list, state) {
  return state.nextIndex < 0 || list.requests?.[state.nextIndex]?.uniqueKey === state.nextUniqueKey;
}

function hasValidStateShape(state) {
  return typeof state?.nextIndex === 'number' && typeof state?.nextUniqueKey === 'string' && Array.isArray(state?.inProgress);
}

try {
  await requestList.initialize();
} catch (err) {
  if (err.message.includes('not consistent with RequestList')) {
    // clear persisted state and retry once with fresh state
    await purgeCachedStorage({ name: 'default' });
    await requestList.initialize();
  } else throw err;
}

When it happens

Trigger: Calling await requestList.initialize() with persistStateKey/persistRequestsKey where the underlying requests source (list of URLs, requestsFromUrl resource, or order of addRequests calls) was modified, reordered, or shortened since the state was persisted; passing a state object obtained from a different RequestList instance.

Common situations: Reordering entries in a remote URL list file consumed via requestsFromUrl; editing a hardcoded URL array between deploys; restoring state saved by a different crawler version or different list; concurrent runs sharing a persisted state key.

Related errors

• The state object is not consistent with RequestList. Unknown
• Cannot clear the state persisted under key '${this.#persistS
• Cannot persist state. options.persistStateKey is not set.
• The state object is invalid: nextIndex must be a non-negativ
• The state object is not consistent with RequestList, too few

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/375f8143920fa58f. Report an issue: GitHub.