apache/flink · critical · IOException

Corrupt data, magic number mismatch. Expected %8x, found %8x

Error message

Corrupt data, magic number mismatch. Expected %8x, found %8x

What it means

Thrown by CompositeTypeSerializerSnapshot.internalReadOuterSnapshot when the first integer read from the serialized snapshot data does not equal the expected MAGIC_NUMBER constant. This is an integrity guard: the binary stream being read is either corrupt, truncated, or not a valid CompositeTypeSerializerSnapshot outer-snapshot segment. It protects against silent mis-deserialization of malformed checkpoint/savepoint data.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeTypeSerializerSnapshot.java:391

        return OuterSchemaCompatibility.COMPATIBLE_AS_IS;
    }

    // ------------------------------------------------------------------------------------------
    //  Utilities
    // ------------------------------------------------------------------------------------------

    private void internalWriteOuterSnapshot(DataOutputView out) throws IOException {
        out.writeInt(MAGIC_NUMBER);
        out.writeInt(getCurrentOuterSnapshotVersion());

        writeOuterSnapshot(out);
    }

    private void internalReadOuterSnapshot(DataInputView in, ClassLoader userCodeClassLoader)
            throws IOException {
        final int magicNumber = in.readInt();
        if (magicNumber != MAGIC_NUMBER) {
            throw new IOException(
                    String.format(
                            "Corrupt data, magic number mismatch. Expected %8x, found %8x",
                            MAGIC_NUMBER, magicNumber));
        }

        final int outerSnapshotVersion = in.readInt();
        readOuterSnapshot(outerSnapshotVersion, in, userCodeClassLoader);
    }

    private void legacyInternalReadOuterSnapshot(
            int legacyReadVersion, DataInputView in, ClassLoader userCodeClassLoader)
            throws IOException {

        // legacy versions did not contain the pre-fixed magic numbers; just read the outer snapshot
        readOuterSnapshot(legacyReadVersion, in, userCodeClassLoader);
    }

    private TypeSerializerSchemaCompatibility<T> constructFinalSchemaCompatibilityResult(

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify the checkpoint/savepoint file is complete and not truncated (check file size against expected metadata).
2. Ensure you are restoring with the same or a compatible Flink version that wrote the checkpoint.
3. If using a custom CompositeTypeSerializerSnapshot subclass, audit its writeOuterSnapshot/readOuterSnapshot pair for symmetry.
4. Re-take the checkpoint from a known-good state if the file is confirmed corrupt.
5. Check storage-layer integrity (S3/HDFS checksums, disk health) if corruption recurs.

// Before restoring, verify the checkpoint file is complete and readable
Path checkpoint = Path.of("/checkpoint/chk-42");
if (!Files.exists(checkpoint) || Files.size(checkpoint) == 0) {
    throw new IllegalStateException("Checkpoint file missing or empty: " + checkpoint);
}

try {
    state.restore(checkpointPath);
} catch (IOException e) {
    if (e.getMessage().contains("magic number mismatch")) {
        // checkpoint data is corrupt or version-incompatible
        log.error("Checkpoint corrupt or incompatible. Re-take from a known-good state.");
    }
    throw e;
}

When it happens

Trigger: Restoring a checkpoint or savepoint whose serialized bytes are partially corrupt (disk error, incomplete write, network truncation). Reading a snapshot stream at the wrong offset (the data input is misaligned). Attempting to deserialize data that was not actually written by internalWriteOuterSnapshot. A custom CompositeTypeSerializerSnapshot subclass whose read/write pair is inconsistent.

Common situations: Corrupted checkpoint files on disk or in object storage. A savepoint taken on one Flink version being read by an incompatible version whose snapshot layout shifted. Concurrent modification or partial flush of state backend files. Misconfigured state backend pointing at stale/partial files.

Related errors

• Corrupt data, magic number mismatch. Expected %8x, found %8x
• Unrecognized version: {}
• Unrecognized version: {}
• Unrecognized version for TypeSerializerSnapshot format: {}
• This object is a dummy TypeSerializer.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/a1072b71b512d9bf. Report an issue: GitHub.