apache/flink · error · IOException

Corrupt data: Unexpected magic number %08X

Error message

Corrupt data: Unexpected magic number %08X

What it means

Thrown by OutputStreamBasedInProgressFileRecoverableSerializer.validateMagicNumber when the first four bytes of the serialized payload do not equal the expected MAGIC_NUMBER constant. The magic number is a guard against deserializing data that was not produced by this serializer — its presence means the byte stream is corrupt or belongs to a different serializer entirely.

Source

Thrown at flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedPartFileWriter.java:446

        }

        private OutputStreamBasedInProgressFileRecoverable deserializeV2(
                final DataInputView dataInputView) throws IOException {
            Path path = null;
            if (dataInputView.readBoolean()) {
                path = new Path(dataInputView.readUTF());
            }
            return new OutputStreamBasedInProgressFileRecoverable(
                    SimpleVersionedSerialization.readVersionAndDeSerialize(
                            resumeSerializer, dataInputView),
                    path);
        }

        private static void validateMagicNumber(final DataInputView dataInputView)
                throws IOException {
            final int magicNumber = dataInputView.readInt();
            if (magicNumber != MAGIC_NUMBER) {
                throw new IOException(
                        String.format("Corrupt data: Unexpected magic number %08X", magicNumber));
            }
        }
    }

    /** The serializer for {@link OutputStreamBasedPendingFileRecoverable}. */
    public static class OutputStreamBasedPendingFileRecoverableSerializer
            implements SimpleVersionedSerializer<PendingFileRecoverable> {

        private static final int MAGIC_NUMBER = 0x2c853c89;

        private final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable>
                commitSerializer;

        OutputStreamBasedPendingFileRecoverableSerializer(
                final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable>
                        commitSerializer) {
            this.commitSerializer = commitSerializer;

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify the checkpoint/savepoint files are intact on the distributed filesystem — check for partial writes, truncation, or corruption.
2. Restart from a known-good checkpoint or savepoint.
3. Ensure the same file sink implementation and Flink version is used for both writing and restoring the checkpoint.

try {
    serializer.deserialize(version, bytes);
} catch (IOException e) {
    if (e.getMessage().contains("Unexpected magic number")) {
        LOG.error("State data is corrupt or from an incompatible serializer");
        // restart from a known-good checkpoint
    }
    throw e;
}

When it happens

Trigger: Deserializing a byte array as an OutputStreamBasedInProgressFileRecoverable where the bytes were produced by a different serializer or truncated/corrupted in storage.

Common situations: Checkpoint or savepoint corruption on the distributed filesystem; state backend data loss or partial writes; using the wrong serializer to read state from a different sink implementation.

Related errors

• Unrecognized version or corrupt state: {version}
• Unrecognized version or corrupt state: {version}
• Unrecognized version or corrupt state: {version}
• Corrupt data: Unexpected magic number %08X
• Bulk Part Writers do not support "pause and resume" operatio

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/30ab718cfecc0580. Report an issue: GitHub.