apify/crawlee · warning · Error

exclusiveStartKey " " was not found in the key-value store…

Error message

exclusiveStartKey "${exclusiveStartKey}" was not found in the key-value store. This is likely a bug — the key may have been deleted between paginated listKeys calls.

What it means

The memory-storage key-value store's listKeys() supports pagination via exclusiveStartKey. If that key is no longer found among filtered items, the library assumes it was deleted between paginated calls and throws, since continuing pagination would be incorrect.

Solutions

Restart listing from the beginning (no exclusiveStartKey) after catching the error
Freeze deletes during pagination or retry the whole listing snapshot
Ensure the exclusiveStartKey and prefix are taken from the same listing call
Serialize writers so keys are not removed mid-pagination

Example fix

// before
const page2 = await store.listKeys({ prefix, exclusiveStartKey: page1.nextExclusiveStartKey });
// after
let page2; try { page2 = await store.listKeys({ prefix, exclusiveStartKey: page1.nextExclusiveStartKey }); } catch { page2 = await store.listKeys({ prefix }); }

Defensive patterns

Strategy: retry

Validate before calling

const first = await store.listKeys({ prefix });
if (first.items.length === 0) return []; // nothing to paginate

Try / catch

try { return await store.listKeys({ prefix, exclusiveStartKey }); } catch (e) { if (String(e).includes('was not found in the key-value store')) return restartListingFromBeginning(); throw e; }

Prevention

Avoid deleting keys while paginating
Always pair exclusiveStartKey with the same prefix used to obtain it
Restart listing from scratch when pagination breaks

When it happens

Trigger: Paginating listKeys() where the record at exclusiveStartKey was deleted (or renamed) between calls, or where a prefix filter excludes the previously returned start key.

Common situations: Long-running crawlers deleting keys while another process lists pages; using a start key obtained with a different prefix; concurrent writers trimming the store during migration.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/7ba3690191fae00c. Report an issue: GitHub.

Appendix: source

Thrown at packages/core/src/memory-storage/resource-clients/key-value-store.ts:131

        for (const record of this.#keyValueEntries.values()) {
            const size = Buffer.byteLength(record.value);
            items.push({
                key: record.key,
                size,
                contentType: record.contentType ?? 'application/octet-stream',
            });
        }

        // Lexically sort to emulate API.
        items.sort((a, b) => a.key.localeCompare(b.key));

        let filteredItems = items.filter((item) => !prefix || item.key.startsWith(prefix));

        if (exclusiveStartKey) {
            const keyPos = filteredItems.findIndex((item) => item.key === exclusiveStartKey);
            if (keyPos === -1) {
                throw new Error(
                    `exclusiveStartKey "${exclusiveStartKey}" was not found in the key-value store. ` +
                        `This is likely a bug — the key may have been deleted between paginated listKeys calls.`,
                );
            }
            filteredItems = filteredItems.slice(keyPos + 1);
        }

        const isTruncated = limit !== undefined && filteredItems.length > limit;
        const pageItems = isTruncated ? filteredItems.slice(0, limit) : filteredItems;
        const nextExclusiveStartKey = isTruncated ? pageItems[pageItems.length - 1].key : undefined;

        this.updateTimestamps(false);

        return {
            items: pageItems,
            count: pageItems.length,
            limit: limit ?? pageItems.length,
            exclusiveStartKey,

View on GitHub (pinned to dbe57fb09c)