apify/crawlee · warning · Error

exclusiveStartKey "${exclusiveStartKey}" was not found in th

Error message

exclusiveStartKey "${exclusiveStartKey}" was not found in the key-value store. This is likely a bug — the key may have been deleted between paginated listKeys calls.

What it means

The memory-storage key-value store's listKeys() supports pagination via exclusiveStartKey. If that key is no longer found among filtered items, the library assumes it was deleted between paginated calls and throws, since continuing pagination would be incorrect.

Source

Thrown at packages/core/src/memory-storage/resource-clients/key-value-store.ts:131

        for (const record of this.#keyValueEntries.values()) {
            const size = Buffer.byteLength(record.value);
            items.push({
                key: record.key,
                size,
                contentType: record.contentType ?? 'application/octet-stream',
            });
        }

        // Lexically sort to emulate API.
        items.sort((a, b) => a.key.localeCompare(b.key));

        let filteredItems = items.filter((item) => !prefix || item.key.startsWith(prefix));

        if (exclusiveStartKey) {
            const keyPos = filteredItems.findIndex((item) => item.key === exclusiveStartKey);
            if (keyPos === -1) {
                throw new Error(
                    `exclusiveStartKey "${exclusiveStartKey}" was not found in the key-value store. ` +
                        `This is likely a bug — the key may have been deleted between paginated listKeys calls.`,
                );
            }
            filteredItems = filteredItems.slice(keyPos + 1);
        }

        const isTruncated = limit !== undefined && filteredItems.length > limit;
        const pageItems = isTruncated ? filteredItems.slice(0, limit) : filteredItems;
        const nextExclusiveStartKey = isTruncated ? pageItems[pageItems.length - 1].key : undefined;

        this.updateTimestamps(false);

        return {
            items: pageItems,
            count: pageItems.length,
            limit: limit ?? pageItems.length,
            exclusiveStartKey,

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Restart listing from the beginning (no exclusiveStartKey) after catching the error
2. Freeze deletes during pagination or retry the whole listing snapshot
3. Ensure the exclusiveStartKey and prefix are taken from the same listing call
4. Serialize writers so keys are not removed mid-pagination

Example fix

// before
const page2 = await store.listKeys({ prefix, exclusiveStartKey: page1.nextExclusiveStartKey });
// after
let page2; try { page2 = await store.listKeys({ prefix, exclusiveStartKey: page1.nextExclusiveStartKey }); } catch { page2 = await store.listKeys({ prefix }); }

const first = await store.listKeys({ prefix });
if (first.items.length === 0) return []; // nothing to paginate

try { return await store.listKeys({ prefix, exclusiveStartKey }); } catch (e) { if (String(e).includes('was not found in the key-value store')) return restartListingFromBeginning(); throw e; }

When it happens

Trigger: Paginating listKeys() where the record at exclusiveStartKey was deleted (or renamed) between calls, or where a prefix filter excludes the previously returned start key.

Common situations: Long-running crawlers deleting keys while another process lists pages; using a start key obtained with a different prefix; concurrent writers trimming the store during migration.

Related errors

• "${segment}" is not allowed because it would resolve outside
• dataset.getData(): The response is too large for parsing. Yo
• The "value" parameter must be a String, Buffer, ArrayBuffer,
• The "value" parameter cannot be stringified to JSON: ${error
• saveSnapshot with key ${key} failed. Cause:${(err as Error).

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/7ba3690191fae00c. Report an issue: GitHub.