{"record":{"id":"a30006e647f5b99d","repo":"apify/crawlee","slug":"dataset-getdata-the-response-is-too-large-for-p","errorCode":null,"errorMessage":"dataset.getData(): The response is too large for parsing. You can fix this by lowering the \"limit\" option.","messagePattern":"dataset\\.getData\\(\\): The response is too large for parsing\\. You can fix this by lowering the \"limit\" option\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/storages/dataset.ts","lineNumber":275,"sourceCode":"                recordedAt: new Date(),\n            });\n            return;\n        }\n\n        this.#statsTracker.add('writeCount');\n        await this.backend.pushData(items);\n    }\n\n    /**\n     * Returns {@apilink DatasetContent} object holding the items in the dataset based on the provided parameters.\n     */\n    async getData(options: DatasetDataOptions = {}): Promise<DatasetContent<Data>> {\n        try {\n            return await this.readPage(options);\n        } catch (e) {\n            const error = e as Error;\n            if (error.message.includes('Cannot create a string longer than')) {\n                throw new Error(\n                    'dataset.getData(): The response is too large for parsing. You can fix this by lowering the \"limit\" option.',\n                );\n            }\n            throw e;\n        }\n    }\n\n    /**\n     * The single transaction-aware page read all dataset read paths go through — both `getData()` and\n     * the private `fetchPages()`. Returns the real page concatenated with the current transaction's\n     * buffered items, with `offset` / `limit` / `desc` windowing applied across the concatenation.\n     */\n    private async readPage(options: DatasetDataOptions): Promise<DatasetContent<Data>> {\n        const buffered = this.bufferedJournalEntries()?.flatMap((entry) => entry.items as Data[]);\n\n        // Every branch below hits the backend exactly once.\n        this.#statsTracker.add('readCount');\n","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/storages/dataset.ts#L257-L293","documentation":"Dataset.getData() wraps readPage() and converts Node's 'Cannot create a string longer than' (V8 max string length, ~512MB/1GB) failures into this clearer error. The requested dataset slice would build a string larger than the runtime limit, so it refuses rather than crashing deep inside JSON parsing.","triggerScenarios":"Calling getData() (or forEachData/ 极 large export paths) on a very large dataset with a huge or default limit so the response exceeds the max string size.","commonSituations":"Dumping an entire multi-GB dataset at once, unbounded limits when paginating, or collecting all results of a long crawl into one getData call.","solutions":["Lower the \"limit\" option and paginate with offset.","Use dataset.forEach() / iterate serially instead of loading everything at once.","Use exportTo/exportToJSON to stream to a KV store file rather than in-memory parsing.","Split the crawl output across multiple datasets to keep each under the size limit."],"exampleFix":"// before\nconst all = await dataset.getData({ limit: 10_000_000 });\n\n// after\nlet offset = 0;\nfor (;;) {\n    const page = await dataset.getData({ limit: 1000, offset });\n    if (!page.items.length) break;\n    process(page.items);\n    offset += page.items.length;\n}","handlingStrategy":"fallback","validationCode":"const { total } = await dataset.getInfo();\nif (total > MAX_PAGE_SIZE) {\n    // paginate instead of a single getData()\n}","typeGuard":null,"tryCatchPattern":"try {\n    return await dataset.getData(opts);\n} catch (e) {\n    if (e instanceof Error && e.message.includes('too large for parsing')) {\n        return await readInPages(dataset, opts, 1000); // paginated fallback\n    }\n    throw e;\n}","preventionTips":["Always set a sane \"limit\" when calling getData() on large datasets.","Prefer dataset.forEach()/pagination over whole-dataset reads.","Use exportTo for bulk dumps instead of in-memory retrieval."],"tags":["dataset","memory","pagination"],"backgroundTag":"response-too-large","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}