apache/flink · error · IOException

Corrupt data: Unexpected magic number ${byteBuffer.getInt()}

Error message

Corrupt data: Unexpected magic number ${byteBuffer.getInt()}

What it means

The v1 OSSRecoverable payload begins with MAGIC_NUMBER. deserializeV1 reads the first int (little-endian); if it differs, the bytes are not a OSSRecoverable v1 body and this IOException with the (already-consumed, hence unreliable) value is thrown. Same corruption signature as 1168 but for the OSS serializer.

Source

Thrown at flink-filesystems/flink-oss-fs-hadoop/src/main/java/org/apache/flink/fs/osshadoop/writer/OSSRecoverableSerializer.java:130

        return resultBytes;
    }

    @Override
    public OSSRecoverable deserialize(int version, byte[] serialized) throws IOException {
        switch (version) {
            case 1:
                return deserializeV1(serialized);
            default:
                throw new IOException("Unrecognized version or corrupt state: " + version);
        }
    }

    private OSSRecoverable deserializeV1(byte[] serialized) throws IOException {
        final ByteBuffer byteBuffer = ByteBuffer.wrap(serialized).order(ByteOrder.LITTLE_ENDIAN);

        if (byteBuffer.getInt() != MAGIC_NUMBER) {
            throw new IOException("Corrupt data: Unexpected magic number " + byteBuffer.getInt());
        }

        final byte[] objectBytes = new byte[byteBuffer.getInt()];
        byteBuffer.get(objectBytes);

        final byte[] uploadIdBytes = new byte[byteBuffer.getInt()];
        byteBuffer.get(uploadIdBytes);

        final int numParts = byteBuffer.getInt();
        final ArrayList<PartETag> parts = new ArrayList<>(numParts);
        for (int i = 0; i < numParts; i++) {
            final int partNum = byteBuffer.getInt();
            final byte[] buffer = new byte[byteBuffer.getInt()];
            byteBuffer.get(buffer);
            parts.add(new PartETag(partNum, new String(buffer, CHARSET)));
        }

        final long numBytes = byteBuffer.getLong();

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure the checkpoint's sink state was produced by the OSS plugin being used at restore (same scheme throughout)
2. Restore from an earlier retained checkpoint if the latest state bytes are corrupt
3. Do not transcode or manually edit serialized recoverable state

null

boolean looksLikeOssRecoverableV1(byte[] bytes) {
    return bytes != null && bytes.length >= 4
        && ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).getInt() == /* MAGIC_NUMBER */ 0x7bab5dea;
}

catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("magic number")) {
        // wrong serializer or corrupt bytes: fail fast; restore an earlier checkpoint if available
    } else { throw e; }
}

When it happens

Trigger: Feeding bytes of a HadoopFsRecoverable or S3 recoverable (or arbitrary corrupt bytes) into OSSRecoverableSerializer.deserialize; truncated state; mixed plugin versions writing/reading the same sink state.

Common situations: Flink downgrade with reused savepoints; switching filesystem plugins under the same sink state; corrupted state backend bytes on OSS/HDFS.

Related errors

• Corrupt data: Unexpected magic number.
• Cannot deserialize and unwrap accumulators properly.
• Failed to serialize ExecutionPlan.
• Failed to deserialize coordination response
• Cannot deserialize and unwrap accumulators properly.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/a31f3ae609faa985. Report an issue: GitHub.