{"record":{"id":"44417b9948725ac6","repo":"apache/flink","slug":"could-not-write-numbytes-bytes-buffer-overflow","errorCode":null,"errorMessage":"Could not write {numBytes} bytes. Buffer overflow.","messagePattern":"Could not write (.+?) bytes\\. Buffer overflow\\.","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/core/memory/DataOutputSerializer.java","lineNumber":385,"sourceCode":"\n        System.arraycopy(this.buffer, 0, nb, 0, this.position);\n        this.buffer = nb;\n        this.wrapper = ByteBuffer.wrap(this.buffer);\n    }\n\n    @Override\n    public void skipBytesToWrite(int numBytes) throws IOException {\n        if (buffer.length - this.position < numBytes) {\n            throw new EOFException(\"Could not skip \" + numBytes + \" bytes.\");\n        }\n\n        this.position += numBytes;\n    }\n\n    @Override\n    public void write(DataInputView source, int numBytes) throws IOException {\n        if (buffer.length - this.position < numBytes) {\n            throw new EOFException(\"Could not write \" + numBytes + \" bytes. Buffer overflow.\");\n        }\n\n        source.readFully(this.buffer, this.position, numBytes);\n        this.position += numBytes;\n    }\n\n    public void setPosition(int position) {\n        Preconditions.checkArgument(\n                position >= 0 && position <= this.position, \"Position out of bounds.\");\n        this.position = position;\n    }\n\n    public void setPositionUnsafe(int position) {\n        this.position = position;\n    }\n\n    // ------------------------------------------------------------------------\n    //  Utilities","sourceCodeStart":367,"sourceCodeEnd":403,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/core/memory/DataOutputSerializer.java#L367-L403","documentation":"DataOutputSerializer is a fixed-capacity byte-array-backed DataOutputView. Its write(DataInputView, int numBytes) copies numBytes bytes from a source view into the internal array and never grows it: if fewer than numBytes bytes remain (buffer.length - position < numBytes) it throws EOFException('Could not write N bytes. Buffer overflow.'). The sibling skipBytesToWrite(int) performs the same capacity check.","triggerScenarios":"Calling DataOutputSerializer.write(DataInputView source, int numBytes) (or skipBytesToWrite) when the serializer's internal array is already filled by prior writes and remaining capacity is smaller than numBytes; typically hit on the copy path where a record is transferred between views with a length prefix larger than the remaining space.","commonSituations":"Reusing a serializer sized for average records across variable-size records; buffers allocated from a length predictor that under-estimates; serialization cutoff tests that write a record bigger than the configured capacity; copying serialized state between DataInputView and DataOutputSerializer during state replication/copy-on-write.","solutions":["Pre-size the DataOutputSerializer for the largest expected record, e.g. new DataOutputSerializer(maxRecordBytes), or reuse one generously sized buffer for the whole copy loop.","Before writing, compare serializer.getAvailableBytes() with numBytes; if insufficient, reset (setPosition(0)/clear) or reallocate a larger serializer.","Catch EOFException around the write and retry with a doubled-capacity serializer (grow-and-retry).","If numBytes comes from a serialized length field, validate it against remaining capacity before calling write."],"exampleFix":"// before\nserializer.write(source, recordLength); // EOFException when recordLength > remaining\n\n// after\nif (serializer.getAvailableBytes() < recordLength) {\n    serializer = new DataOutputSerializer(Math.max(recordLength, 2 * serializer.length()));\n}\nserializer.write(source, recordLength);","handlingStrategy":"validation","validationCode":"// before calling serializer.write(source, numBytes)\nint available = serializer.length() - serializer.getPosition();\nif (available < numBytes) {\n    serializer = new DataOutputSerializer(Math.max(numBytes, 2 * serializer.length()));\n}","typeGuard":null,"tryCatchPattern":"try {\n    serializer.write(source, numBytes);\n} catch (EOFException e) {\n    // grow buffer and retry once, or fail with record-size context\n}","preventionTips":["Size DataOutputSerializer buffers from the worst-case record size, not the average.","Never assume write(DataInputView, int) grows the buffer — it is fixed-capacity by design.","In copy loops, track remaining capacity and reset/reallocate before it hits zero."],"tags":["serialization","memory","flink-core","buffer-overflow"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}