apache/flink · critical · OutOfMemoryError

Required array size too large

Error message

Required array size too large

What it means

An OutOfMemoryError thrown by FileUtils.readAllBytes(Path) when the file is larger than MAX_BUFFER_SIZE (Integer.MAX_VALUE - 8, ~2GB), the largest byte[] Java can allocate. This mirrors java.nio.file.Files.readAllBytes behavior: whole-file reads into one array are structurally capped at ~2GB.

Source

Thrown at flink-core/src/main/java/org/apache/flink/util/FileUtils.java:178

     * <p>This is an implementation that follow {@link
     * java.nio.file.Files#readAllBytes(java.nio.file.Path)}, and the difference is that it limits
     * the size of the direct buffer to avoid direct-buffer OutOfMemoryError. When {@link
     * java.nio.file.Files#readAllBytes(java.nio.file.Path)} or other interfaces in java API can do
     * this in the future, we should remove it.
     *
     * @param path the path to the file
     * @return a byte array containing the bytes read from the file
     * @throws IOException if an I/O error occurs reading from the stream
     * @throws OutOfMemoryError if an array of the required size cannot be allocated, for example
     *     the file is larger that {@code 2GB}
     */
    public static byte[] readAllBytes(java.nio.file.Path path) throws IOException {
        try (SeekableByteChannel channel = Files.newByteChannel(path);
                InputStream in = Channels.newInputStream(channel)) {

            long size = channel.size();
            if (size > (long) MAX_BUFFER_SIZE) {
                throw new OutOfMemoryError("Required array size too large");
            }

            return read(in, (int) size);
        }
    }

    /**
     * Reads all the bytes from an input stream. Uses {@code initialSize} as a hint about how many
     * bytes the stream will have and uses {@code directBufferSize} to limit the size of the direct
     * buffer used to read.
     *
     * @param source the input stream to read from
     * @param initialSize the initial size of the byte array to allocate
     * @return a byte array containing the bytes read from the file
     * @throws IOException if an I/O error occurs reading from the stream
     * @throws OutOfMemoryError if an array of the required size cannot be allocated
     */
    public static byte[] read(InputStream source, int initialSize) throws IOException {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Stream the file instead of loading it whole: use InputStream + IOUtils.copyBytes, or memory-mapped/SeekableByteChannel processing in chunks
2. If the file should never be that big, find why it is (wrong path, concatenated logs, corrupted blob) and fix the producer
3. Raise heap only as a last resort — the 2GB array cap makes full read impossible regardless of -Xmx

Example fix

// before
byte[] all = FileUtils.readAllBytes(path); // >2GB file -> OOM error

// after
try (InputStream in = Files.newInputStream(path)) {
    // process in chunks, never materialize the whole file
    byte[] chunk = new byte[8 * 1024 * 1024];
    for (int n; (n = in.read(chunk)) > 0;) { consume(chunk, n); }
}

long size = Files.size(path);
if (size > Integer.MAX_VALUE - 8) {
    throw new IOException("File too large to read into memory (" + size + " bytes): " + path);
}

catch (OutOfMemoryError e) and convert to a domain error naming the file and its size; never retry — the size is deterministic.

When it happens

Trigger: Calling FileUtils.readAllBytes on a file whose channel.size() exceeds ~2GB — e.g. loading a huge JAR/blob/log file fully into memory. The error is thrown up front based on the file size, before reading starts.

Common situations: Reading large session blobs, heap-dump-like artifacts, or misconfigured inputs that point at a giant file; 32-bit or small-heap JVMs where even sizes below 2GB fail array allocation (that surfaces as a plain OutOfMemoryError from the allocator instead).

Related errors

• DistributedCache supports only local files for Collection En
• Failed to serialize element. Serialized size (> {newLen} byt
• Could not write {numBytes} bytes. Buffer overflow.
• MemorySegment can be freed only once!
• Memory segment does not represent heap memory

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/4ce332288b13a234. Report an issue: GitHub.