What causes "Corrupt data: Unexpected magic number." in apache/flink?

Ensure the byte array passed to deserialize is exactly the output of LocalRecoverableSerializer.serialize — no extra framing bytes prepended or appended.

apache/flink · error · IOException

Corrupt data: Unexpected magic number.

Error message

Corrupt data: Unexpected magic number.

What it means

Thrown inside LocalRecoverableSerializer.deserializeV1 when the first four bytes of the deserialized payload (read in LITTLE_ENDIAN) do not equal the expected MAGIC_NUMBER constant (0x1e744b57). The magic number is a sanity sentinel written at serialize time to confirm the byte stream is genuinely a LocalRecoverable payload and not random or mis-framed data.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableSerializer.java:80

        return targetBytes;
    }

    @Override
    public LocalRecoverable deserialize(int version, byte[] serialized) throws IOException {
        switch (version) {
            case 1:
                return deserializeV1(serialized);
            default:
                throw new IOException("Unrecognized version or corrupt state: " + version);
        }
    }

    private static LocalRecoverable deserializeV1(byte[] serialized) throws IOException {
        final ByteBuffer bb = ByteBuffer.wrap(serialized).order(ByteOrder.LITTLE_ENDIAN);

        if (bb.getInt() != MAGIC_NUMBER) {
            throw new IOException("Corrupt data: Unexpected magic number.");
        }

        final long offset = bb.getLong();
        final byte[] targetFileBytes = new byte[bb.getInt()];
        final byte[] tempFileBytes = new byte[bb.getInt()];
        bb.get(targetFileBytes);
        bb.get(tempFileBytes);

        final String targetPath = new String(targetFileBytes, CHARSET);
        final String tempPath = new String(tempFileBytes, CHARSET);

        return new LocalRecoverable(new File(targetPath), new File(tempPath), offset);
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

Ensure the byte array passed to deserialize is exactly the output of LocalRecoverableSerializer.serialize — no extra framing bytes prepended or appended.
If reading from a SimpleVersionedSerialization stream, use readVersionAndDeSerialize which strips the version+length header automatically, rather than calling deserialize directly.
Verify the checkpoint file is not corrupted by checking its size against expected metadata.
If the data is genuinely from a different source, use the correct serializer for that recoverable type.

Example fix

// before — raw bytes fed directly, header not stripped
byte[] raw = Files.readAllBytes(metaPath);
LocalRecoverable r = LocalRecoverableSerializer.INSTANCE.deserialize(1, raw);

// after — use the standard framing utility
try (DataInputViewStreamWrapper in = new DataInputViewStreamWrapper(Files.newInputStream(metaPath))) {
    LocalRecoverable r = SimpleVersionedSerialization.readVersionAndDeSerialize(
        LocalRecoverableSerializer.INSTANCE, in);
}

Defensive patterns

Strategy: try-catch

Validate before calling

ByteBuffer probe = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN);
if (probe.remaining() < 4 || probe.getInt() != 0x1e744b57) {
    throw new IOException("Not a valid LocalRecoverable payload (bad magic number)");
}

Try / catch

try {
    LocalRecoverable r = LocalRecoverableSerializer.INSTANCE.deserialize(1, bytes);
} catch (IOException e) {
    if (e.getMessage().contains("magic number")) {
        // payload is not LocalRecoverable data; use correct serializer or regenerate
    }
    throw e;
}

Prevention

Use SimpleVersionedSerialization framing consistently so bytes are never fed raw to deserialize.
Never manually slice or prepend bytes to a serialized LocalRecoverable payload.
Store recovery metadata with its full framing to avoid offset errors.

When it happens

Trigger: Calling LocalRecoverableSerializer.deserialize(1, bytes) where bytes were not produced by the matching serialize() method; passing a byte array whose first four bytes were truncated, shifted, or belong to a different serializer's format; accidentally feeding the full SimpleVersionedSerialization framing (version+length header) directly to deserialize instead of stripping the 8-byte header first.

Common situations: Checkpoint/savepoint corruption on local disk or in-memory transport; manual byte-array manipulation or incorrect offsets when slicing serialized data; version field says 1 but the payload came from a different recoverable type (e.g. HadoopRecoverable).

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/1fb5a7936049b92c. Report an issue: GitHub.