{"record":{"id":"db9c2340d5d13119","repo":"prestodb/presto","slug":"can-not-grow-array-beyond-s","errorCode":null,"errorMessage":"Can not grow array beyond '%s'","messagePattern":"Can not grow array beyond '(.+?)'","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"presto-common/src/main/java/com/facebook/presto/common/block/BlockUtil.java","lineNumber":99,"sourceCode":"        //sourceIndex + length can overflow integer range\n        if (sourceIndex > MAX_ARRAY_SIZE - length) {\n            throw new SliceTooLargeException(format(\"Cannot allocate slice larger than %d bytes\", MAX_ARRAY_SIZE));\n        }\n    }\n\n    static int calculateNewArraySize(int currentSize)\n    {\n        // grow array by 50%\n        long newSize = (long) currentSize + (currentSize >> 1);\n\n        // verify new size is within reasonable bounds\n        if (newSize < DEFAULT_CAPACITY) {\n            newSize = DEFAULT_CAPACITY;\n        }\n        else if (newSize > MAX_ARRAY_SIZE) {\n            newSize = MAX_ARRAY_SIZE;\n            if (newSize == currentSize) {\n                throw new IllegalArgumentException(format(\"Can not grow array beyond '%s'\", MAX_ARRAY_SIZE));\n            }\n        }\n        return (int) newSize;\n    }\n\n    static int calculateBlockResetSize(int currentSize)\n    {\n        long newSize = (long) ceil(currentSize * BLOCK_RESET_SKEW);\n\n        // verify new size is within reasonable bounds\n        if (newSize < DEFAULT_CAPACITY) {\n            newSize = DEFAULT_CAPACITY;\n        }\n        else if (newSize > MAX_ARRAY_SIZE) {\n            newSize = MAX_ARRAY_SIZE;\n        }\n        return (int) newSize;\n    }","sourceCodeStart":81,"sourceCodeEnd":117,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-common/src/main/java/com/facebook/presto/common/block/BlockUtil.java#L81-L117","documentation":"Thrown by BlockUtil.calculateNewArraySize when a block's internal backing array must be grown but the requested size already equals MAX_ARRAY_SIZE, so no further growth is possible. The library caps all block arrays at MAX_ARRAY_SIZE to avoid overflowing int-indexed arrays and exhausting heap, and this error signals that hard ceiling has been reached. It is a capacity-planning failure, not data corruption.","triggerScenarios":"Calling calculateNewArraySize (directly or via block builder growth paths) with a currentSize already equal to MAX_ARRAY_SIZE where the computed new size clamps back to MAX_ARRAY_SIZE; i.e., attempting to append to a block builder whose array is already at the maximum allowed size.","commonSituations":"Building an extremely large single block (aggregations, big scans, large VALUES lists) that exceeds the JVM's practical per-array limit; misconfigured aggregation memory limits allowing one builder to grow unbounded; 32-bit indexing limits hit on very large datasets.","solutions":["Reduce the size of the data going into a single block/builder: process data in smaller batches or partitions so no single builder reaches MAX_ARRAY_SIZE.","Increase task/-operator memory limits only if growth is legitimate, and let spillable operations stream results instead of materializing one huge block.","Check upstream logic for runaway growth (a builder never flushed/reset); ensure blocks are periodically flushed to pages instead of accumulating.","If you call calculateNewArraySize yourself, check currentSize >= MAX_ARRAY_SIZE before growing and handle the case explicitly."],"exampleFix":"// before\nBlockBuilder builder = ...;\nfor (Row row : billionsOfRows) {\n    builder.appendRow(row); // eventually hits MAX_ARRAY_SIZE\n}\n// after\nList<Page> pages = new ArrayList<>();\nBlockBuilder builder = ...;\nfor (Row row : billionsOfRows) {\n    builder.appendRow(row);\n    if (builder.getPositionCount() >= TARGET_PAGE_SIZE) {\n        pages.add(builder.build());\n        builder = builder.newBlockBuilderLike(null); // reset before hitting cap\n    }\n}\npages.add(builder.build());","handlingStrategy":"validation","validationCode":"// before appending more data to a builder/block-backed array\nint nextSize = BlockUtil.calculateNewArraySize(currentSize);\nif (currentSize >= Integer.MAX_VALUE - 8 || nextSize <= currentSize) {\n    throw new IllegalStateException(\"block array cannot grow further; flush or partition data\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    builder.appendXxx(value);\n} catch (IllegalArgumentException e) {\n    if (e.getMessage() != null && e.getMessage().startsWith(\"Can not grow array beyond\")) {\n        flushCurrentBlockAndStartNewBuilder(); // recover by partitioning\n    } else {\n        throw e;\n    }\n}","preventionTips":["Flush block builders into pages at a fixed position-count threshold instead of accumulating unbounded data.","Size operator/aggregation memory limits so a single builder cannot reach MAX_ARRAY_SIZE.","If calling calculateNewArraySize directly, check currentSize against MAX_ARRAY_SIZE first.","Prefer streaming/spillable operators over materializing one giant block."],"tags":["presto","block","array-capacity","memory-limit"],"backgroundTag":"array-size-limit-exceeded","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}