{"record":{"id":"fd459400aee0f08a","repo":"apache/flink","slug":"corrupt-data-unexpected-magic-number-08x","errorCode":null,"errorMessage":"Corrupt data: Unexpected magic number %08X","messagePattern":"Corrupt data: Unexpected magic number %08X","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSinkCommittableSerializer.java","lineNumber":161,"sourceCode":"        if (dataInputView.readBoolean()) {\n            inProgressFileToCleanup =\n                    SimpleVersionedSerialization.readVersionAndDeSerialize(\n                            inProgressFileSerializer, dataInputView);\n        }\n\n        Path committedFileToCleanup = null;\n        if (dataInputView.readBoolean()) {\n            committedFileToCleanup = new Path(dataInputView.readUTF());\n        }\n\n        return new FileSinkCommittable(\n                bucketId, pendingFile, inProgressFileToCleanup, committedFileToCleanup);\n    }\n\n    private static void validateMagicNumber(DataInputView in) throws IOException {\n        int magicNumber = in.readInt();\n        if (magicNumber != MAGIC_NUMBER) {\n            throw new IOException(\n                    String.format(\"Corrupt data: Unexpected magic number %08X\", magicNumber));\n        }\n    }\n}\n","sourceCodeStart":143,"sourceCodeEnd":166,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSinkCommittableSerializer.java#L143-L166","documentation":"Thrown by validateMagicNumber when the first 4 bytes of the serialized data do not match the expected magic number 0x1e765c80. This integrity check catches cases where the byte array being deserialized is not actually a FileSinkCommittable, has been truncated, or is corrupt. The error message includes the actual magic number found, formatted as hex.","triggerScenarios":"Deserializing arbitrary or mismatched byte data as a FileSinkCommittable; state data corrupted in storage (disk error, S3 consistency issue); wrong serializer used to read the data; truncated or partial state records.","commonSituations":"State backend corruption; checkpoint restore from a savepoint produced by a different sink type; partial writes to the checkpoint store; storage-level data integrity failure.","solutions":["Verify the checkpoint/savepoint was produced by a FileSink with the compatible committable serializer.","Check for storage corruption (disk errors, S3 eventual consistency issues, network-level data corruption).","If state is genuinely incompatible or corrupt, start fresh without restoring state.","Inspect the hex magic number in the error to determine if the data is from a different serializer entirely."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify data integrity before deserialization\nDataInputDeserializer in = new DataInputDeserializer(data);\nint magicNumber = in.readInt();\nif (magicNumber != 0x1e765c80) {\n    throw new IOException(\"Data is not a valid FileSinkCommittable (wrong magic number)\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    serializer.deserialize(version, data);\n} catch (IOException e) {\n    if (e.getMessage().startsWith(\"Corrupt data: Unexpected magic number\")) {\n        // data is not a valid FileSinkCommittable or is corrupt\n        // check storage integrity and source of the data\n    }\n    throw e;\n}","preventionTips":["Verify checkpoint storage integrity (disk health, S3 consistency).","Ensure the savepoint was produced by a FileSink with the same serializer.","Do not manually modify or truncate state data."],"tags":["file-sink","serialization","data-corruption","state-restore","integrity-check"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}