apache/flink · error · IOException

Corrupt data: Unexpected magic number %08X

Error message

Corrupt data: Unexpected magic number %08X

What it means

Thrown by validateMagicNumber when the first 4 bytes of the serialized data do not match the expected magic number 0x1e765c80. This integrity check catches cases where the byte array being deserialized is not actually a FileSinkCommittable, has been truncated, or is corrupt. The error message includes the actual magic number found, formatted as hex.

Source

Thrown at flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSinkCommittableSerializer.java:161

        if (dataInputView.readBoolean()) {
            inProgressFileToCleanup =
                    SimpleVersionedSerialization.readVersionAndDeSerialize(
                            inProgressFileSerializer, dataInputView);
        }

        Path committedFileToCleanup = null;
        if (dataInputView.readBoolean()) {
            committedFileToCleanup = new Path(dataInputView.readUTF());
        }

        return new FileSinkCommittable(
                bucketId, pendingFile, inProgressFileToCleanup, committedFileToCleanup);
    }

    private static void validateMagicNumber(DataInputView in) throws IOException {
        int magicNumber = in.readInt();
        if (magicNumber != MAGIC_NUMBER) {
            throw new IOException(
                    String.format("Corrupt data: Unexpected magic number %08X", magicNumber));
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify the checkpoint/savepoint was produced by a FileSink with the compatible committable serializer.
2. Check for storage corruption (disk errors, S3 eventual consistency issues, network-level data corruption).
3. If state is genuinely incompatible or corrupt, start fresh without restoring state.
4. Inspect the hex magic number in the error to determine if the data is from a different serializer entirely.

// Verify data integrity before deserialization
DataInputDeserializer in = new DataInputDeserializer(data);
int magicNumber = in.readInt();
if (magicNumber != 0x1e765c80) {
    throw new IOException("Data is not a valid FileSinkCommittable (wrong magic number)");
}

try {
    serializer.deserialize(version, data);
} catch (IOException e) {
    if (e.getMessage().startsWith("Corrupt data: Unexpected magic number")) {
        // data is not a valid FileSinkCommittable or is corrupt
        // check storage integrity and source of the data
    }
    throw e;
}

When it happens

Trigger: Deserializing arbitrary or mismatched byte data as a FileSinkCommittable; state data corrupted in storage (disk error, S3 consistency issue); wrong serializer used to read the data; truncated or partial state records.

Common situations: State backend corruption; checkpoint restore from a savepoint produced by a different sink type; partial writes to the checkpoint store; storage-level data integrity failure.

Related errors

• Unrecognized version or corrupt state: {version}
• Committable to compact has no content.
• Unrecognized version or corrupt state: {version}
• Corrupt data: Unexpected magic number %08X
• Unrecognized version or corrupt state: {version}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/fd459400aee0f08a. Report an issue: GitHub.