apify/crawlee · error · Error

The state object is not consistent with RequestList, too few

Error message

The state object is not consistent with RequestList, too few requests loaded.

What it means

Beyond being well-formed, the restored nextIndex must not exceed the number of loaded requests — otherwise the list cannot resume where it left off. This error means the persisted state references more requests than the current sources produced (state.nextIndex > this.requests.length).

Source

Thrown at packages/core/src/storages/request_list.ts:507

    private async persistRequests(): Promise<void> {
        const serializedRequests = await serializeArray(this.requests);
        this.#store ??= await KeyValueStore.open();
        await this.#store.setValue(this.#persistRequestsKey!, serializedRequests, { contentType: CONTENT_TYPE_BINARY });
        this.areRequestsPersisted = true;
    }

    /**
     * Restores RequestList state from a state object.
     */
    private restoreState(state?: RequestListState): void {
        // If there's no state it means we've not persisted any (yet).
        if (!state) return;
        // Restore previous state.
        if (typeof state.nextIndex !== 'number' || state.nextIndex < 0) {
            throw new Error('The state object is invalid: nextIndex must be a non-negative number.');
        }
        if (state.nextIndex > this.requests.length) {
            throw new Error('The state object is not consistent with RequestList, too few requests loaded.');
        }
        if (
            state.nextIndex < this.requests.length &&
            this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey
        ) {
            throw new Error(
                'The state object is not consistent with RequestList the order of URLs seems to have changed.',
            );
        }

        const deleteFromInProgress: string[] = [];
        state.inProgress.forEach((uniqueKey) => {
            const index = this.#uniqueKeyToIndex[uniqueKey];
            if (typeof index !== 'number') {
                throw new Error(
                    'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',
                );
            }

View on GitHub (pinned to dbe57fb09c)

Solutions

Align sources so the list contains at least nextIndex requests (restore the removed URLs)
Delete the persisted state record to start fresh, or use a new persistStateKey for the changed list
Verify sourcesFunction returns the same/complete set of URLs as the run that persisted the state

Example fix

// before
// state { nextIndex: 500 } but sources now produce 300 requests
// after
const store = await KeyValueStore.open();
await store.setValue('CRAWLEE_list-state', null); // reset state to match the new sources
// or restore the full sources array so length >= nextIndex

Defensive patterns

Strategy: try-catch

Validate before calling

const stored = await store.getValue('CRAWLEE_list-state');
if (stored && typeof stored.nextIndex === 'number' && sources && stored.nextIndex > sources.length) {
  await store.setValue('CRAWLEE_list-state', null); // state belongs to a larger source set — reset
}

Try / catch

try {
  const list = await RequestList.open('list', { sources, persistStateKey: 'state' });
} catch (e) {
  if (e.message.includes('too few requests loaded')) {
    const store = await KeyValueStore.open();
    await store.setValue('CRAWLEE_state', null);
    return RequestList.open('list', { sources, persistStateKey: 'state' });
  }
  throw e;
}

Prevention

Keep the sources list stable across runs while state is persisted
Use a distinct persistStateKey whenever you materially change sources
Clear persisted state when intentionally shrinking the source list

When it happens

Trigger: Re-running with fewer/smaller sources than the previous run (sources array trimmed, sourcesFunction returning fewer URLs), while a state persisted from the larger run is restored; switching persistStateKey between lists so state belongs to a different request set.

Common situations: Editing the sources list between deployments without clearing persisted state; a dynamic sourcesFunction returning fewer items due to an upstream API change; reusing the same persistStateKey for a different RequestList.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/f8c3581875053777. Report an issue: GitHub.