apache/flink · error · IOException

Corrupt data, conflicting lengths. Length fields: {}, data:

Error message

Corrupt data, conflicting lengths. Length fields: {}, data: {}

What it means

Thrown by SimpleVersionedSerialization.readVersionAndDeSerialize(serializer, byte[]) when the length field embedded in the framing header (bytes 4-7, big-endian) does not equal the actual remaining byte count (bytes 8..end). The writer records the serialized datum length in the header so the reader can cross-check; a mismatch indicates the byte array was truncated, extended, or is not valid SimpleVersionedSerialization output.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerialization.java:229

        checkArgument(bytes.length >= 8, "byte array below minimum length (8 bytes)");

        final byte[] dataOnly = Arrays.copyOfRange(bytes, 8, bytes.length);
        final int version =
                ((bytes[0] & 0xff) << 24)
                        | ((bytes[1] & 0xff) << 16)
                        | ((bytes[2] & 0xff) << 8)
                        | (bytes[3] & 0xff);

        final int length =
                ((bytes[4] & 0xff) << 24)
                        | ((bytes[5] & 0xff) << 16)
                        | ((bytes[6] & 0xff) << 8)
                        | (bytes[7] & 0xff);

        if (length == dataOnly.length) {
            return serializer.deserialize(version, dataOnly);
        } else {
            throw new IOException(
                    "Corrupt data, conflicting lengths. Length fields: "
                            + length
                            + ", data: "
                            + dataOnly.length);
        }
    }

    // ------------------------------------------------------------------------

    /** Utility class, not meant to be instantiated. */
    private SimpleVersionedSerialization() {}
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure the byte array was produced by SimpleVersionedSerialization.writeVersionAndSerialize and has not been modified.
2. If you only have raw serialized bytes (no framing), call serializer.deserialize(version, rawBytes) directly instead of readVersionAndDeSerialize.
3. Verify storage/transport does not truncate the byte array — check that the stored length matches the original.
4. Add a length check: bytes.length must be >= 8 and the header length field must equal bytes.length - 8.

Example fix

// before — raw serializer bytes fed to framed reader
byte[] raw = mySerializer.serialize(datum);
T result = SimpleVersionedSerialization.readVersionAndDeSerialize(mySerializer, raw);

// after — use the matching writer to frame, or read raw directly
byte[] framed = SimpleVersionedSerialization.writeVersionAndSerialize(mySerializer, datum);
T result = SimpleVersionedSerialization.readVersionAndDeSerialize(mySerializer, framed);

if (bytes.length < 8) throw new IOException("Below minimum framing length");
int len = ((bytes[4]&0xff)<<24)|((bytes[5]&0xff)<<16)|((bytes[6]&0xff)<<8)|(bytes[7]&0xff);
if (len != bytes.length - 8) throw new IOException("Length field mismatch");

try {
    T result = SimpleVersionedSerialization.readVersionAndDeSerialize(serializer, bytes);
} catch (IOException e) {
    if (e.getMessage().contains("conflicting lengths")) {
        // data is not framed output; use serializer.deserialize directly if raw
    }
    throw e;
}

When it happens

Trigger: Passing a byte array to readVersionAndDeSerialize that was not produced by the matching writeVersionAndSerialize; truncation or corruption of the byte array after serialization; manually concatenating or slicing serialized payloads without preserving framing.

Common situations: Corrupted state metadata in checkpoints/savepoints; network transport truncation; storing serialized bytes in a column with a length limit; feeding a raw serializer output (without the 8-byte version+length header) to readVersionAndDeSerialize.

Related errors

• Corrupt data: Unexpected magic number %08X
• Corrupt data, magic number mismatch. Expected %8x, found %8x
• Corrupt data, magic number mismatch. Expected %8x, found %8x
• Corrupt data: Unexpected magic number.
• PostVersionedIOReadableWritable cannot read from a DataInput

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/eecd4acc421fd9d1. Report an issue: GitHub.