apache/flink · error · EOFException

Could not write {numBytes} bytes. Buffer overflow.

Error message

Could not write {numBytes} bytes. Buffer overflow.

What it means

DataOutputSerializer is a fixed-capacity byte-array-backed DataOutputView. Its write(DataInputView, int numBytes) copies numBytes bytes from a source view into the internal array and never grows it: if fewer than numBytes bytes remain (buffer.length - position < numBytes) it throws EOFException('Could not write N bytes. Buffer overflow.'). The sibling skipBytesToWrite(int) performs the same capacity check.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/memory/DataOutputSerializer.java:385

        System.arraycopy(this.buffer, 0, nb, 0, this.position);
        this.buffer = nb;
        this.wrapper = ByteBuffer.wrap(this.buffer);
    }

    @Override
    public void skipBytesToWrite(int numBytes) throws IOException {
        if (buffer.length - this.position < numBytes) {
            throw new EOFException("Could not skip " + numBytes + " bytes.");
        }

        this.position += numBytes;
    }

    @Override
    public void write(DataInputView source, int numBytes) throws IOException {
        if (buffer.length - this.position < numBytes) {
            throw new EOFException("Could not write " + numBytes + " bytes. Buffer overflow.");
        }

        source.readFully(this.buffer, this.position, numBytes);
        this.position += numBytes;
    }

    public void setPosition(int position) {
        Preconditions.checkArgument(
                position >= 0 && position <= this.position, "Position out of bounds.");
        this.position = position;
    }

    public void setPositionUnsafe(int position) {
        this.position = position;
    }

    // ------------------------------------------------------------------------
    //  Utilities

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Pre-size the DataOutputSerializer for the largest expected record, e.g. new DataOutputSerializer(maxRecordBytes), or reuse one generously sized buffer for the whole copy loop.
2. Before writing, compare serializer.getAvailableBytes() with numBytes; if insufficient, reset (setPosition(0)/clear) or reallocate a larger serializer.
3. Catch EOFException around the write and retry with a doubled-capacity serializer (grow-and-retry).
4. If numBytes comes from a serialized length field, validate it against remaining capacity before calling write.

Example fix

// before
serializer.write(source, recordLength); // EOFException when recordLength > remaining

// after
if (serializer.getAvailableBytes() < recordLength) {
    serializer = new DataOutputSerializer(Math.max(recordLength, 2 * serializer.length()));
}
serializer.write(source, recordLength);

// before calling serializer.write(source, numBytes)
int available = serializer.length() - serializer.getPosition();
if (available < numBytes) {
    serializer = new DataOutputSerializer(Math.max(numBytes, 2 * serializer.length()));
}

try {
    serializer.write(source, numBytes);
} catch (EOFException e) {
    // grow buffer and retry once, or fail with record-size context
}

When it happens

Trigger: Calling DataOutputSerializer.write(DataInputView source, int numBytes) (or skipBytesToWrite) when the serializer's internal array is already filled by prior writes and remaining capacity is smaller than numBytes; typically hit on the copy path where a record is transferred between views with a length prefix larger than the remaining space.

Common situations: Reusing a serializer sized for average records across variable-size records; buffers allocated from a length predictor that under-estimates; serialization cutoff tests that write a record bigger than the configured capacity; copying serialized state between DataInputView and DataOutputSerializer during state replication/copy-on-write.

Related errors

• The Kryo Output still contains data from a previous serializ
• Error during Java serialization.
• Error during Java deserialization.
• Could not clone serializer instance of class {className}
• Unrecognized type. This method is deprecated and might not w

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/44417b9948725ac6. Report an issue: GitHub.