{"record":{"id":"eecd4acc421fd9d1","repo":"apache/flink","slug":"corrupt-data-conflicting-lengths-length-fields","errorCode":null,"errorMessage":"Corrupt data, conflicting lengths. Length fields: {}, data: {}","messagePattern":"Corrupt data, conflicting lengths\\. Length fields: (.+?), data: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerialization.java","lineNumber":229,"sourceCode":"        checkArgument(bytes.length >= 8, \"byte array below minimum length (8 bytes)\");\n\n        final byte[] dataOnly = Arrays.copyOfRange(bytes, 8, bytes.length);\n        final int version =\n                ((bytes[0] & 0xff) << 24)\n                        | ((bytes[1] & 0xff) << 16)\n                        | ((bytes[2] & 0xff) << 8)\n                        | (bytes[3] & 0xff);\n\n        final int length =\n                ((bytes[4] & 0xff) << 24)\n                        | ((bytes[5] & 0xff) << 16)\n                        | ((bytes[6] & 0xff) << 8)\n                        | (bytes[7] & 0xff);\n\n        if (length == dataOnly.length) {\n            return serializer.deserialize(version, dataOnly);\n        } else {\n            throw new IOException(\n                    \"Corrupt data, conflicting lengths. Length fields: \"\n                            + length\n                            + \", data: \"\n                            + dataOnly.length);\n        }\n    }\n\n    // ------------------------------------------------------------------------\n\n    /** Utility class, not meant to be instantiated. */\n    private SimpleVersionedSerialization() {}\n}\n","sourceCodeStart":211,"sourceCodeEnd":242,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerialization.java#L211-L242","documentation":"Thrown by SimpleVersionedSerialization.readVersionAndDeSerialize(serializer, byte[]) when the length field embedded in the framing header (bytes 4-7, big-endian) does not equal the actual remaining byte count (bytes 8..end). The writer records the serialized datum length in the header so the reader can cross-check; a mismatch indicates the byte array was truncated, extended, or is not valid SimpleVersionedSerialization output.","triggerScenarios":"Passing a byte array to readVersionAndDeSerialize that was not produced by the matching writeVersionAndSerialize; truncation or corruption of the byte array after serialization; manually concatenating or slicing serialized payloads without preserving framing.","commonSituations":"Corrupted state metadata in checkpoints/savepoints; network transport truncation; storing serialized bytes in a column with a length limit; feeding a raw serializer output (without the 8-byte version+length header) to readVersionAndDeSerialize.","solutions":["Ensure the byte array was produced by SimpleVersionedSerialization.writeVersionAndSerialize and has not been modified.","If you only have raw serialized bytes (no framing), call serializer.deserialize(version, rawBytes) directly instead of readVersionAndDeSerialize.","Verify storage/transport does not truncate the byte array — check that the stored length matches the original.","Add a length check: bytes.length must be >= 8 and the header length field must equal bytes.length - 8."],"exampleFix":"// before — raw serializer bytes fed to framed reader\nbyte[] raw = mySerializer.serialize(datum);\nT result = SimpleVersionedSerialization.readVersionAndDeSerialize(mySerializer, raw);\n\n// after — use the matching writer to frame, or read raw directly\nbyte[] framed = SimpleVersionedSerialization.writeVersionAndSerialize(mySerializer, datum);\nT result = SimpleVersionedSerialization.readVersionAndDeSerialize(mySerializer, framed);","handlingStrategy":"validation","validationCode":"if (bytes.length < 8) throw new IOException(\"Below minimum framing length\");\nint len = ((bytes[4]&0xff)<<24)|((bytes[5]&0xff)<<16)|((bytes[6]&0xff)<<8)|(bytes[7]&0xff);\nif (len != bytes.length - 8) throw new IOException(\"Length field mismatch\");","typeGuard":null,"tryCatchPattern":"try {\n    T result = SimpleVersionedSerialization.readVersionAndDeSerialize(serializer, bytes);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"conflicting lengths\")) {\n        // data is not framed output; use serializer.deserialize directly if raw\n    }\n    throw e;\n}","preventionTips":["Always pair writeVersionAndSerialize with readVersionAndDeSerialize.","Do not strip or add bytes to framed payloads.","Verify storage preserves full byte length."],"tags":["serialization","data-corruption","framing","versioned-io"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}