prestodb/presto · error · IllegalArgumentException

Can not grow array beyond '%s'

Error message

Can not grow array beyond '%s'

What it means

Thrown by BlockUtil.calculateNewArraySize when a block's internal backing array must be grown but the requested size already equals MAX_ARRAY_SIZE, so no further growth is possible. The library caps all block arrays at MAX_ARRAY_SIZE to avoid overflowing int-indexed arrays and exhausting heap, and this error signals that hard ceiling has been reached. It is a capacity-planning failure, not data corruption.

Source

Thrown at presto-common/src/main/java/com/facebook/presto/common/block/BlockUtil.java:99

        //sourceIndex + length can overflow integer range
        if (sourceIndex > MAX_ARRAY_SIZE - length) {
            throw new SliceTooLargeException(format("Cannot allocate slice larger than %d bytes", MAX_ARRAY_SIZE));
        }
    }

    static int calculateNewArraySize(int currentSize)
    {
        // grow array by 50%
        long newSize = (long) currentSize + (currentSize >> 1);

        // verify new size is within reasonable bounds
        if (newSize < DEFAULT_CAPACITY) {
            newSize = DEFAULT_CAPACITY;
        }
        else if (newSize > MAX_ARRAY_SIZE) {
            newSize = MAX_ARRAY_SIZE;
            if (newSize == currentSize) {
                throw new IllegalArgumentException(format("Can not grow array beyond '%s'", MAX_ARRAY_SIZE));
            }
        }
        return (int) newSize;
    }

    static int calculateBlockResetSize(int currentSize)
    {
        long newSize = (long) ceil(currentSize * BLOCK_RESET_SKEW);

        // verify new size is within reasonable bounds
        if (newSize < DEFAULT_CAPACITY) {
            newSize = DEFAULT_CAPACITY;
        }
        else if (newSize > MAX_ARRAY_SIZE) {
            newSize = MAX_ARRAY_SIZE;
        }
        return (int) newSize;
    }

View on GitHub (pinned to 55bb57d202)

Solutions

1. Reduce the size of the data going into a single block/builder: process data in smaller batches or partitions so no single builder reaches MAX_ARRAY_SIZE.
2. Increase task/-operator memory limits only if growth is legitimate, and let spillable operations stream results instead of materializing one huge block.
3. Check upstream logic for runaway growth (a builder never flushed/reset); ensure blocks are periodically flushed to pages instead of accumulating.
4. If you call calculateNewArraySize yourself, check currentSize >= MAX_ARRAY_SIZE before growing and handle the case explicitly.

Example fix

// before
BlockBuilder builder = ...;
for (Row row : billionsOfRows) {
    builder.appendRow(row); // eventually hits MAX_ARRAY_SIZE
}
// after
List<Page> pages = new ArrayList<>();
BlockBuilder builder = ...;
for (Row row : billionsOfRows) {
    builder.appendRow(row);
    if (builder.getPositionCount() >= TARGET_PAGE_SIZE) {
        pages.add(builder.build());
        builder = builder.newBlockBuilderLike(null); // reset before hitting cap
    }
}
pages.add(builder.build());

// before appending more data to a builder/block-backed array
int nextSize = BlockUtil.calculateNewArraySize(currentSize);
if (currentSize >= Integer.MAX_VALUE - 8 || nextSize <= currentSize) {
    throw new IllegalStateException("block array cannot grow further; flush or partition data");
}

try {
    builder.appendXxx(value);
} catch (IllegalArgumentException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("Can not grow array beyond")) {
        flushCurrentBlockAndStartNewBuilder(); // recover by partitioning
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling calculateNewArraySize (directly or via block builder growth paths) with a currentSize already equal to MAX_ARRAY_SIZE where the computed new size clamps back to MAX_ARRAY_SIZE; i.e., attempting to append to a block builder whose array is already at the maximum allowed size.

Common situations: Building an extremely large single block (aggregations, big scans, large VALUES lists) that exceeds the JVM's practical per-array limit; misconfigured aggregation memory limits allowing one builder to grow unbounded; 32-bit indexing limits hit on very large datasets.

Related errors

• New column does not have same number of rows as old column
• Declared positions (%s) does not match block %s's number of
• Offset is not monotonically ascending. offsets[%s]=%s, offse
• A null map must have zero entries
• Number of fields in RowBlock must be positive

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/db9c2340d5d13119. Report an issue: GitHub.