apache/flink · error · EOFException

Could not skip {numBytes} bytes.

Error message

Could not skip {numBytes} bytes.

What it means

Thrown by DataOutputSerializer.skipBytesToWrite(int) when the remaining buffer (buffer.length - position) is less than numBytes. Unlike write(), skipBytesToWrite does NOT auto-resize the buffer, so it is a hard failure when there is insufficient space. It is typically used to reserve placeholder bytes (e.g. a length prefix) that are back-filled later.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/memory/DataOutputSerializer.java:376

                }
            } else {
                throw new IOException(
                        "Failed to serialize element. Serialized size (> "
                                + newLen
                                + " bytes) exceeds JVM heap space",
                        e);
            }
        }

        System.arraycopy(this.buffer, 0, nb, 0, this.position);
        this.buffer = nb;
        this.wrapper = ByteBuffer.wrap(this.buffer);
    }

    @Override
    public void skipBytesToWrite(int numBytes) throws IOException {
        if (buffer.length - this.position < numBytes) {
            throw new EOFException("Could not skip " + numBytes + " bytes.");
        }

        this.position += numBytes;
    }

    @Override
    public void write(DataInputView source, int numBytes) throws IOException {
        if (buffer.length - this.position < numBytes) {
            throw new EOFException("Could not write " + numBytes + " bytes. Buffer overflow.");
        }

        source.readFully(this.buffer, this.position, numBytes);
        this.position += numBytes;
    }

    public void setPosition(int position) {
        Preconditions.checkArgument(
                position >= 0 && position <= this.position, "Position out of bounds.");

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure the buffer can hold position + numBytes before skipBytesToWrite (size the DataOutputSerializer generously or write first to trigger auto-resize).
2. Use write(source, n) or explicit writes (which auto-resize) instead of skipBytesToWrite when the content size is unknown.
3. Reserve the prefix slot before writing content so position stays within capacity.

Example fix

// before
out.skipBytesToWrite(4); // reserve length prefix, may overflow
// ... write content ...

// after: reserve prefix first while buffer is fresh
out.skipBytesToWrite(4);
int lenPos = out.length() - 4;
// ... write content (auto-resizes) ...
out.writeIntUnsafe(recordLen, lenPos); // back-fill

// skipBytesToWrite does NOT auto-resize; ensure capacity first
// Trigger an auto-resize by writing a placeholder, or size the serializer generously
if (out.length() + numBytes > /* known capacity */ Integer.MAX_VALUE) {
    throw new EOFException("Not enough buffer to skip " + numBytes + " bytes");
}

When it happens

Trigger: Calling skipBytesToWrite(n) when buffer.length - position < n; commonly when reserving a length-prefix slot without first ensuring capacity.

Common situations: Back-patching a length field after writing content but on a too-small or fixed buffer; using skipBytesToWrite instead of write on a buffer sized only for the content; forgetting that skip does not grow the buffer.

Related errors

• The Kryo Output still contains data from a previous serializ
• Invalid bounds.
• Length may not be negative.
• Could not skip {numBytes} bytes.
• Offset cannot be negative.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/0c5b20792fc47f2e. Report an issue: GitHub.