{"record":{"id":"9642dcefc3aa95fd","repo":"apache/flink","slug":"serialization-failed-because-the-record-length-wou","errorCode":null,"errorMessage":"Serialization failed because the record length would exceed 2GB (max addressable array size in Java).","messagePattern":"Serialization failed because the record length would exceed 2GB \\(max addressable array size in Java\\)\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"flink-core/src/main/java/org/apache/flink/core/memory/DataOutputSerializer.java","lineNumber":342,"sourceCode":"    }\n\n    private int getUTFBytesSize(int c) {\n        if ((c >= 0x0001) && (c <= 0x007F)) {\n            return 1;\n        } else if (c > 0x07FF) {\n            return 3;\n        } else {\n            return 2;\n        }\n    }\n\n    private void resize(int minCapacityAdd) throws IOException {\n        int newLen = Math.max(this.buffer.length * 2, this.buffer.length + minCapacityAdd);\n        byte[] nb;\n        try {\n            nb = new byte[newLen];\n        } catch (NegativeArraySizeException e) {\n            throw new IOException(\n                    \"Serialization failed because the record length would exceed 2GB (max addressable array size in Java).\");\n        } catch (OutOfMemoryError e) {\n            // this was too large to allocate, try the smaller size (if possible)\n            if (newLen > this.buffer.length + minCapacityAdd) {\n                newLen = this.buffer.length + minCapacityAdd;\n                try {\n                    nb = new byte[newLen];\n                } catch (OutOfMemoryError ee) {\n                    // still not possible. give an informative exception message that reports the\n                    // size\n                    throw new IOException(\n                            \"Failed to serialize element. Serialized size (> \"\n                                    + newLen\n                                    + \" bytes) exceeds JVM heap space\",\n                            ee);\n                }\n            } else {\n                throw new IOException(","sourceCodeStart":324,"sourceCodeEnd":360,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/core/memory/DataOutputSerializer.java#L324-L360","documentation":"Thrown by DataOutputSerializer.resize(int) when doubling the internal buffer produced a newLen that overflowed Java's int (going negative), causing new byte[newLen] to throw NegativeArraySizeException. It is wrapped as an IOException indicating the serialized record would exceed the ~2GB maximum addressable Java array size.","triggerScenarios":"Serializing a single record whose cumulative size forces the buffer past Integer.MAX_VALUE (~2.1GB), making buffer.length*2 overflow to a negative int.","commonSituations":"Serializing huge objects: large byte[]/String payloads, deeply nested collections, or unbounded accumulation in a single DataOutputSerializer instance without clear().","solutions":["Avoid serializing single records larger than ~2GB; chunk or stream them instead.","Call clear() and flush intermediate data for serializers that accumulate across many records.","Enforce a per-record size cap at the application boundary."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// No cheap pre-check; cap record size at the application boundary\nstatic final int MAX_RECORD_BYTES = 1 << 30; // 1 GiB safety cap\nif (estimatedRecordSize > MAX_RECORD_BYTES) {\n    throw new IllegalArgumentException(\"Record too large to serialize: \" + estimatedRecordSize);\n}","typeGuard":null,"tryCatchPattern":"try {\n    out.writeLongUTF(huge);\n} catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"2GB\")) {\n        throw new IllegalArgumentException(\"Record exceeds 2GB serialization limit\", e);\n    }\n    throw e;\n}","preventionTips":["Never serialize a single record larger than ~2GB; chunk or stream it.","Call clear() on reused DataOutputSerializer instances between records.","Enforce a per-record size cap at the application boundary."],"tags":["serialization","size-limit","memory","array-overflow"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}