apify/crawlee · error · Error

The state object is not consistent with RequestList, too…

Error message

The state object is not consistent with RequestList, too few requests loaded.

What it means

Beyond being well-formed, the restored nextIndex must not exceed the number of loaded requests — otherwise the list cannot resume where it left off. This error means the persisted state references more requests than the current sources produced (state.nextIndex > this.requests.length).

Solutions

Align sources so the list contains at least nextIndex requests (restore the removed URLs)
Delete the persisted state record to start fresh, or use a new persistStateKey for the changed list
Verify sourcesFunction returns the same/complete set of URLs as the run that persisted the state

Example fix

// before
// state { nextIndex: 500 } but sources now produce 300 requests
// after
const store = await KeyValueStore.open();
await store.setValue('CRAWLEE_list-state', null); // reset state to match the new sources
// or restore the full sources array so length >= nextIndex

Defensive patterns

Strategy: try-catch

Validate before calling

const stored = await store.getValue('CRAWLEE_list-state');
if (stored && typeof stored.nextIndex === 'number' && sources && stored.nextIndex > sources.length) {
  await store.setValue('CRAWLEE_list-state', null); // state belongs to a larger source set — reset
}

Try / catch

try {
  const list = await RequestList.open('list', { sources, persistStateKey: 'state' });
} catch (e) {
  if (e.message.includes('too few requests loaded')) {
    const store = await KeyValueStore.open();
    await store.setValue('CRAWLEE_state', null);
    return RequestList.open('list', { sources, persistStateKey: 'state' });
  }
  throw e;
}

Prevention

Keep the sources list stable across runs while state is persisted
Use a distinct persistStateKey whenever you materially change sources
Clear persisted state when intentionally shrinking the source list

When it happens

Trigger: Re-running with fewer/smaller sources than the previous run (sources array trimmed, sourcesFunction returning fewer URLs), while a state persisted from the larger run is restored; switching persistStateKey between lists so state belongs to a different request set.

Common situations: Editing the sources list between deployments without clearing persisted state; a dynamic sourcesFunction returning fewer items due to an upstream API change; reusing the same persistStateKey for a different RequestList.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/f8c3581875053777. Report an issue: GitHub.

Appendix: source

Thrown at packages/core/src/storages/request_list.ts:507

    private async persistRequests(): Promise<void> {
        const serializedRequests = await serializeArray(this.requests);
        this.#store ??= await KeyValueStore.open();
        await this.#store.setValue(this.#persistRequestsKey!, serializedRequests, { contentType: CONTENT_TYPE_BINARY });
        this.areRequestsPersisted = true;
    }

    /**
     * Restores RequestList state from a state object.
     */
    private restoreState(state?: RequestListState): void {
        // If there's no state it means we've not persisted any (yet).
        if (!state) return;
        // Restore previous state.
        if (typeof state.nextIndex !== 'number' || state.nextIndex < 0) {
            throw new Error('The state object is invalid: nextIndex must be a non-negative number.');
        }
        if (state.nextIndex > this.requests.length) {
            throw new Error('The state object is not consistent with RequestList, too few requests loaded.');
        }
        if (
            state.nextIndex < this.requests.length &&
            this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey
        ) {
            throw new Error(
                'The state object is not consistent with RequestList the order of URLs seems to have changed.',
            );
        }

        const deleteFromInProgress: string[] = [];
        state.inProgress.forEach((uniqueKey) => {
            const index = this.#uniqueKeyToIndex[uniqueKey];
            if (typeof index !== 'number') {
                throw new Error(
                    'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',
                );
            }

View on GitHub (pinned to dbe57fb09c)