{"record":{"id":"30ab718cfecc0580","repo":"apache/flink","slug":"corrupt-data-unexpected-magic-number-08x-30ab71","errorCode":null,"errorMessage":"Corrupt data: Unexpected magic number %08X","messagePattern":"Corrupt data: Unexpected magic number %08X","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedPartFileWriter.java","lineNumber":446,"sourceCode":"        }\n\n        private OutputStreamBasedInProgressFileRecoverable deserializeV2(\n                final DataInputView dataInputView) throws IOException {\n            Path path = null;\n            if (dataInputView.readBoolean()) {\n                path = new Path(dataInputView.readUTF());\n            }\n            return new OutputStreamBasedInProgressFileRecoverable(\n                    SimpleVersionedSerialization.readVersionAndDeSerialize(\n                            resumeSerializer, dataInputView),\n                    path);\n        }\n\n        private static void validateMagicNumber(final DataInputView dataInputView)\n                throws IOException {\n            final int magicNumber = dataInputView.readInt();\n            if (magicNumber != MAGIC_NUMBER) {\n                throw new IOException(\n                        String.format(\"Corrupt data: Unexpected magic number %08X\", magicNumber));\n            }\n        }\n    }\n\n    /** The serializer for {@link OutputStreamBasedPendingFileRecoverable}. */\n    public static class OutputStreamBasedPendingFileRecoverableSerializer\n            implements SimpleVersionedSerializer<PendingFileRecoverable> {\n\n        private static final int MAGIC_NUMBER = 0x2c853c89;\n\n        private final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable>\n                commitSerializer;\n\n        OutputStreamBasedPendingFileRecoverableSerializer(\n                final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable>\n                        commitSerializer) {\n            this.commitSerializer = commitSerializer;","sourceCodeStart":428,"sourceCodeEnd":464,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedPartFileWriter.java#L428-L464","documentation":"Thrown by OutputStreamBasedInProgressFileRecoverableSerializer.validateMagicNumber when the first four bytes of the serialized payload do not equal the expected MAGIC_NUMBER constant. The magic number is a guard against deserializing data that was not produced by this serializer — its presence means the byte stream is corrupt or belongs to a different serializer entirely.","triggerScenarios":"Deserializing a byte array as an OutputStreamBasedInProgressFileRecoverable where the bytes were produced by a different serializer or truncated/corrupted in storage.","commonSituations":"Checkpoint or savepoint corruption on the distributed filesystem; state backend data loss or partial writes; using the wrong serializer to read state from a different sink implementation.","solutions":["Verify the checkpoint/savepoint files are intact on the distributed filesystem — check for partial writes, truncation, or corruption.","Restart from a known-good checkpoint or savepoint.","Ensure the same file sink implementation and Flink version is used for both writing and restoring the checkpoint."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    serializer.deserialize(version, bytes);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"Unexpected magic number\")) {\n        LOG.error(\"State data is corrupt or from an incompatible serializer\");\n        // restart from a known-good checkpoint\n    }\n    throw e;\n}","preventionTips":["Verify checkpoint/savepoint file integrity on the distributed filesystem.","Use the same serializer and Flink version for write and restore.","Monitor storage layer health for partial writes or data loss."],"tags":["file-sink","serialization","corruption","checkpoint"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}