apache/flink · critical · IOException

Serialization failed because the record length would exceed

Error message

Serialization failed because the record length would exceed 2GB (max addressable array size in Java).

What it means

Thrown by DataOutputSerializer.resize(int) when doubling the internal buffer produced a newLen that overflowed Java's int (going negative), causing new byte[newLen] to throw NegativeArraySizeException. It is wrapped as an IOException indicating the serialized record would exceed the ~2GB maximum addressable Java array size.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/memory/DataOutputSerializer.java:342

    }

    private int getUTFBytesSize(int c) {
        if ((c >= 0x0001) && (c <= 0x007F)) {
            return 1;
        } else if (c > 0x07FF) {
            return 3;
        } else {
            return 2;
        }
    }

    private void resize(int minCapacityAdd) throws IOException {
        int newLen = Math.max(this.buffer.length * 2, this.buffer.length + minCapacityAdd);
        byte[] nb;
        try {
            nb = new byte[newLen];
        } catch (NegativeArraySizeException e) {
            throw new IOException(
                    "Serialization failed because the record length would exceed 2GB (max addressable array size in Java).");
        } catch (OutOfMemoryError e) {
            // this was too large to allocate, try the smaller size (if possible)
            if (newLen > this.buffer.length + minCapacityAdd) {
                newLen = this.buffer.length + minCapacityAdd;
                try {
                    nb = new byte[newLen];
                } catch (OutOfMemoryError ee) {
                    // still not possible. give an informative exception message that reports the
                    // size
                    throw new IOException(
                            "Failed to serialize element. Serialized size (> "
                                    + newLen
                                    + " bytes) exceeds JVM heap space",
                            ee);
                }
            } else {
                throw new IOException(

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Avoid serializing single records larger than ~2GB; chunk or stream them instead.
2. Call clear() and flush intermediate data for serializers that accumulate across many records.
3. Enforce a per-record size cap at the application boundary.

// No cheap pre-check; cap record size at the application boundary
static final int MAX_RECORD_BYTES = 1 << 30; // 1 GiB safety cap
if (estimatedRecordSize > MAX_RECORD_BYTES) {
    throw new IllegalArgumentException("Record too large to serialize: " + estimatedRecordSize);
}

try {
    out.writeLongUTF(huge);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("2GB")) {
        throw new IllegalArgumentException("Record exceeds 2GB serialization limit", e);
    }
    throw e;
}

When it happens

Trigger: Serializing a single record whose cumulative size forces the buffer past Integer.MAX_VALUE (~2.1GB), making buffer.length*2 overflow to a negative int.

Common situations: Serializing huge objects: large byte[]/String payloads, deeply nested collections, or unbounded accumulation in a single DataOutputSerializer instance without clear().

Related errors

• Encoded string reached maximum length: {utflen}
• encoded string too long: {} memory
• Encoded string is too long: {utflen}
• Failed to serialize element. Serialized size (> {newLen} byt
• Could not write {numBytes} bytes. Buffer overflow.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/9642dcefc3aa95fd. Report an issue: GitHub.