{"record":{"id":"4ce332288b13a234","repo":"apache/flink","slug":"required-array-size-too-large","errorCode":null,"errorMessage":"Required array size too large","messagePattern":"Required array size too large","errorType":"panic","errorClass":"OutOfMemoryError","httpStatus":null,"severity":"critical","filePath":"flink-core/src/main/java/org/apache/flink/util/FileUtils.java","lineNumber":178,"sourceCode":"     * <p>This is an implementation that follow {@link\n     * java.nio.file.Files#readAllBytes(java.nio.file.Path)}, and the difference is that it limits\n     * the size of the direct buffer to avoid direct-buffer OutOfMemoryError. When {@link\n     * java.nio.file.Files#readAllBytes(java.nio.file.Path)} or other interfaces in java API can do\n     * this in the future, we should remove it.\n     *\n     * @param path the path to the file\n     * @return a byte array containing the bytes read from the file\n     * @throws IOException if an I/O error occurs reading from the stream\n     * @throws OutOfMemoryError if an array of the required size cannot be allocated, for example\n     *     the file is larger that {@code 2GB}\n     */\n    public static byte[] readAllBytes(java.nio.file.Path path) throws IOException {\n        try (SeekableByteChannel channel = Files.newByteChannel(path);\n                InputStream in = Channels.newInputStream(channel)) {\n\n            long size = channel.size();\n            if (size > (long) MAX_BUFFER_SIZE) {\n                throw new OutOfMemoryError(\"Required array size too large\");\n            }\n\n            return read(in, (int) size);\n        }\n    }\n\n    /**\n     * Reads all the bytes from an input stream. Uses {@code initialSize} as a hint about how many\n     * bytes the stream will have and uses {@code directBufferSize} to limit the size of the direct\n     * buffer used to read.\n     *\n     * @param source the input stream to read from\n     * @param initialSize the initial size of the byte array to allocate\n     * @return a byte array containing the bytes read from the file\n     * @throws IOException if an I/O error occurs reading from the stream\n     * @throws OutOfMemoryError if an array of the required size cannot be allocated\n     */\n    public static byte[] read(InputStream source, int initialSize) throws IOException {","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L160-L196","documentation":"An OutOfMemoryError thrown by FileUtils.readAllBytes(Path) when the file is larger than MAX_BUFFER_SIZE (Integer.MAX_VALUE - 8, ~2GB), the largest byte[] Java can allocate. This mirrors java.nio.file.Files.readAllBytes behavior: whole-file reads into one array are structurally capped at ~2GB.","triggerScenarios":"Calling FileUtils.readAllBytes on a file whose channel.size() exceeds ~2GB — e.g. loading a huge JAR/blob/log file fully into memory. The error is thrown up front based on the file size, before reading starts.","commonSituations":"Reading large session blobs, heap-dump-like artifacts, or misconfigured inputs that point at a giant file; 32-bit or small-heap JVMs where even sizes below 2GB fail array allocation (that surfaces as a plain OutOfMemoryError from the allocator instead).","solutions":["Stream the file instead of loading it whole: use InputStream + IOUtils.copyBytes, or memory-mapped/SeekableByteChannel processing in chunks","If the file should never be that big, find why it is (wrong path, concatenated logs, corrupted blob) and fix the producer","Raise heap only as a last resort — the 2GB array cap makes full read impossible regardless of -Xmx"],"exampleFix":"// before\nbyte[] all = FileUtils.readAllBytes(path); // >2GB file -> OOM error\n\n// after\ntry (InputStream in = Files.newInputStream(path)) {\n    // process in chunks, never materialize the whole file\n    byte[] chunk = new byte[8 * 1024 * 1024];\n    for (int n; (n = in.read(chunk)) > 0;) { consume(chunk, n); }\n}","handlingStrategy":"validation","validationCode":"long size = Files.size(path);\nif (size > Integer.MAX_VALUE - 8) {\n    throw new IOException(\"File too large to read into memory (\" + size + \" bytes): \" + path);\n}","typeGuard":null,"tryCatchPattern":"catch (OutOfMemoryError e) and convert to a domain error naming the file and its size; never retry — the size is deterministic.","preventionTips":["Stream large files chunk-wise instead of whole-file byte[] reads","Add size guards at boundaries that accept uploaded/downloaded files"],"tags":["flink-core","filesystem","memory","large-files","out-of-memory"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}