apify/crawlee · error · Error

dataset.getData(): The response is too large for parsing. Yo

Error message

dataset.getData(): The response is too large for parsing. You can fix this by lowering the "limit" option.

What it means

Dataset.getData() wraps readPage() and converts Node's 'Cannot create a string longer than' (V8 max string length, ~512MB/1GB) failures into this clearer error. The requested dataset slice would build a string larger than the runtime limit, so it refuses rather than crashing deep inside JSON parsing.

Source

Thrown at packages/core/src/storages/dataset.ts:275

                recordedAt: new Date(),
            });
            return;
        }

        this.#statsTracker.add('writeCount');
        await this.backend.pushData(items);
    }

    /**
     * Returns {@apilink DatasetContent} object holding the items in the dataset based on the provided parameters.
     */
    async getData(options: DatasetDataOptions = {}): Promise<DatasetContent<Data>> {
        try {
            return await this.readPage(options);
        } catch (e) {
            const error = e as Error;
            if (error.message.includes('Cannot create a string longer than')) {
                throw new Error(
                    'dataset.getData(): The response is too large for parsing. You can fix this by lowering the "limit" option.',
                );
            }
            throw e;
        }
    }

    /**
     * The single transaction-aware page read all dataset read paths go through — both `getData()` and
     * the private `fetchPages()`. Returns the real page concatenated with the current transaction's
     * buffered items, with `offset` / `limit` / `desc` windowing applied across the concatenation.
     */
    private async readPage(options: DatasetDataOptions): Promise<DatasetContent<Data>> {
        const buffered = this.bufferedJournalEntries()?.flatMap((entry) => entry.items as Data[]);

        // Every branch below hits the backend exactly once.
        this.#statsTracker.add('readCount');

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Lower the "limit" option and paginate with offset.
2. Use dataset.forEach() / iterate serially instead of loading everything at once.
3. Use exportTo/exportToJSON to stream to a KV store file rather than in-memory parsing.
4. Split the crawl output across multiple datasets to keep each under the size limit.

Example fix

// before
const all = await dataset.getData({ limit: 10_000_000 });

// after
let offset = 0;
for (;;) {
    const page = await dataset.getData({ limit: 1000, offset });
    if (!page.items.length) break;
    process(page.items);
    offset += page.items.length;
}

const { total } = await dataset.getInfo();
if (total > MAX_PAGE_SIZE) {
    // paginate instead of a single getData()
}

try {
    return await dataset.getData(opts);
} catch (e) {
    if (e instanceof Error && e.message.includes('too large for parsing')) {
        return await readInPages(dataset, opts, 1000); // paginated fallback
    }
    throw e;
}

When it happens

Trigger: Calling getData() (or forEachData/ 极 large export paths) on a very large dataset with a huge or default limit so the response exceeds the max string size.

Common situations: Dumping an entire multi-GB dataset at once, unbounded limits when paginating, or collecting all results of a long crawl into one getData call.

Related errors

• availableMemoryRatio is not set in configuration.
• exclusiveStartKey "${exclusiveStartKey}" was not found in th
• Data item${s}is not an object. You can push only objects int
• Data item${s}is not serializable to JSON. Cause: ${err.messa
• Unsupported content type: ${contentType}

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/a30006e647f5b99d. Report an issue: GitHub.