apache/flink · error · IllegalStateException

The Kryo Output still contains data from a previous serializ

Error message

The Kryo Output still contains data from a previous serialize call. It has to be flushed or cleared at the end of the serialize call.

What it means

KryoSerializer.serialize() reuses one Kryo Output object across calls and requires it to be empty at entry. If output.position() != 0 it means a previous serialize call ended without flushing (typically one that failed with an EOFException/buffer-full and left bytes behind), so writing again would duplicate data. Flink throws IllegalStateException as a sanity guard telling you the Output must be flushed or cleared.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java:340

    @Override
    public void serialize(T record, DataOutputView target) throws IOException {
        if (CONCURRENT_ACCESS_CHECK) {
            enterExclusiveThread();
        }

        try {
            checkKryoInitialized();

            if (target != previousOut) {
                DataOutputViewStream outputStream = new DataOutputViewStream(target);
                output = new Output(outputStream);
                previousOut = target;
            }

            // Sanity check: Make sure that the output is cleared/has been flushed by the last call
            // otherwise data might be written multiple times in case of a previous EOFException
            if (output.position() != 0) {
                throw new IllegalStateException(
                        "The Kryo Output still contains data from a previous "
                                + "serialize call. It has to be flushed or cleared at the end of the serialize call.");
            }

            try {
                kryo.writeClassAndObject(output, record);
                output.flush();
            } catch (KryoException ke) {
                // make sure that the Kryo output buffer is reset in case that we can recover from
                // the exception (e.g. EOFException which denotes buffer full)
                output.reset();

                Throwable cause = ke.getCause();
                if (cause instanceof EOFException) {
                    throw (EOFException) cause;
                } else {
                    throw ke;
                }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Treat the original failure as the root cause: find the earlier KryoException (often EOFException 'Buffer overflow') in the logs and fix it (raise the buffer size or shrink the record).
2. Increase memory-segment-related settings so a single record fits: raise taskmanager.memory.segment-size or the network buffer budget so the bounded output can hold the record.
3. If a custom Kryo serializer is involved, ensure it does not throw after partially writing, and that it does not call output.flush() prematurely.
4. Reduce record size (project only needed fields, or split the record) so Kryo serialization never hits the buffer limit.

Example fix

// before: record larger than the target buffer causes prior EOFException, then
// the next serialize() hits 'Output still contains data'
DataStream<Row> stream = env.fromElements(hugeRow);

// after: give the network more room (flink-conf.yaml)
// taskmanager.memory.segment-size: 32kb -> 1mb
// or shrink the record before serialization:
DataStream<SmallRow> stream = env.fromElements(hugeRow).map(r -> project(r));

try {
    serializer.serialize(record, target);
} catch (IllegalStateException e) {
    if (e.getMessage().contains("still contains data")) {
    }
    throw e;
}

When it happens

Trigger: A previous kryo.writeClassAndObject threw a KryoException whose data partially remained in the buffer (the catch calls output.reset(), but exceptions escaping between flush points can leave residue); a serializer implementation or intercepted write path that skips the flush; serialization into a fixed-size target buffer that hit EOF on the prior record and the recovery reset did not run.

Common situations: Serializing large records into bounded buffers (network or intermediate results) where the previous record hit a buffer overflow; custom Kryo serializers that write but do not let the outer flush complete; races or re-entrant calls into the same KryoSerializer instance after a prior failure.

Related errors

• Error during Java serialization.
• Error during Java deserialization.
• Could not clone serializer instance of class {className}
• Could not write {numBytes} bytes. Buffer overflow.
• Failed to serialize value '{value}'

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/58519529ee52c425. Report an issue: GitHub.