{"record":{"id":"a1072b71b512d9bf","repo":"apache/flink","slug":"corrupt-data-magic-number-mismatch-expected-8x","errorCode":null,"errorMessage":"Corrupt data, magic number mismatch. Expected %8x, found %8x","messagePattern":"Corrupt data, magic number mismatch\\. Expected %8x, found %8x","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeTypeSerializerSnapshot.java","lineNumber":391,"sourceCode":"        return OuterSchemaCompatibility.COMPATIBLE_AS_IS;\n    }\n\n    // ------------------------------------------------------------------------------------------\n    //  Utilities\n    // ------------------------------------------------------------------------------------------\n\n    private void internalWriteOuterSnapshot(DataOutputView out) throws IOException {\n        out.writeInt(MAGIC_NUMBER);\n        out.writeInt(getCurrentOuterSnapshotVersion());\n\n        writeOuterSnapshot(out);\n    }\n\n    private void internalReadOuterSnapshot(DataInputView in, ClassLoader userCodeClassLoader)\n            throws IOException {\n        final int magicNumber = in.readInt();\n        if (magicNumber != MAGIC_NUMBER) {\n            throw new IOException(\n                    String.format(\n                            \"Corrupt data, magic number mismatch. Expected %8x, found %8x\",\n                            MAGIC_NUMBER, magicNumber));\n        }\n\n        final int outerSnapshotVersion = in.readInt();\n        readOuterSnapshot(outerSnapshotVersion, in, userCodeClassLoader);\n    }\n\n    private void legacyInternalReadOuterSnapshot(\n            int legacyReadVersion, DataInputView in, ClassLoader userCodeClassLoader)\n            throws IOException {\n\n        // legacy versions did not contain the pre-fixed magic numbers; just read the outer snapshot\n        readOuterSnapshot(legacyReadVersion, in, userCodeClassLoader);\n    }\n\n    private TypeSerializerSchemaCompatibility<T> constructFinalSchemaCompatibilityResult(","sourceCodeStart":373,"sourceCodeEnd":409,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeTypeSerializerSnapshot.java#L373-L409","documentation":"Thrown by CompositeTypeSerializerSnapshot.internalReadOuterSnapshot when the first integer read from the serialized snapshot data does not equal the expected MAGIC_NUMBER constant. This is an integrity guard: the binary stream being read is either corrupt, truncated, or not a valid CompositeTypeSerializerSnapshot outer-snapshot segment. It protects against silent mis-deserialization of malformed checkpoint/savepoint data.","triggerScenarios":"Restoring a checkpoint or savepoint whose serialized bytes are partially corrupt (disk error, incomplete write, network truncation). Reading a snapshot stream at the wrong offset (the data input is misaligned). Attempting to deserialize data that was not actually written by internalWriteOuterSnapshot. A custom CompositeTypeSerializerSnapshot subclass whose read/write pair is inconsistent.","commonSituations":"Corrupted checkpoint files on disk or in object storage. A savepoint taken on one Flink version being read by an incompatible version whose snapshot layout shifted. Concurrent modification or partial flush of state backend files. Misconfigured state backend pointing at stale/partial files.","solutions":["Verify the checkpoint/savepoint file is complete and not truncated (check file size against expected metadata).","Ensure you are restoring with the same or a compatible Flink version that wrote the checkpoint.","If using a custom CompositeTypeSerializerSnapshot subclass, audit its writeOuterSnapshot/readOuterSnapshot pair for symmetry.","Re-take the checkpoint from a known-good state if the file is confirmed corrupt.","Check storage-layer integrity (S3/HDFS checksums, disk health) if corruption recurs."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before restoring, verify the checkpoint file is complete and readable\nPath checkpoint = Path.of(\"/checkpoint/chk-42\");\nif (!Files.exists(checkpoint) || Files.size(checkpoint) == 0) {\n    throw new IllegalStateException(\"Checkpoint file missing or empty: \" + checkpoint);\n}","typeGuard":null,"tryCatchPattern":"try {\n    state.restore(checkpointPath);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"magic number mismatch\")) {\n        // checkpoint data is corrupt or version-incompatible\n        log.error(\"Checkpoint corrupt or incompatible. Re-take from a known-good state.\");\n    }\n    throw e;\n}","preventionTips":["Always confirm checkpoints/savepoints are fully written before relying on them (check _metadata completeness).","Keep Flink versions aligned between write and restore, or follow the documented upgrade path.","Use durable storage with checksums (HDFS/S3) and verify integrity after transfer.","Audit custom CompositeTypeSerializerSnapshot subclasses for symmetric write/read pairs."],"tags":["serialization","checkpoint","savepoint","data-corruption","state","io"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}