apache/flink · error · IOException

Corrupt data: Unexpected magic number.

Error message

Corrupt data: Unexpected magic number.

What it means

Error "Corrupt data: Unexpected magic number." thrown in apache/flink.

Source

Thrown at flink-formats/flink-hadoop-bulk/src/main/java/org/apache/flink/formats/hadoop/bulk/HadoopPathBasedPartFileWriter.java:218

        @Override
        public HadoopPathBasedPendingFileRecoverable deserialize(int version, byte[] serialized)
                throws IOException {
            switch (version) {
                case 1:
                    return deserializeV1(serialized);
                case 2:
                    return deserializeV2(serialized);
                default:
                    throw new IOException("Unrecognized version or corrupt state: " + version);
            }
        }

        private HadoopPathBasedPendingFileRecoverable deserializeV1(byte[] serialized)
                throws IOException {
            final ByteBuffer bb = ByteBuffer.wrap(serialized).order(ByteOrder.LITTLE_ENDIAN);

            if (bb.getInt() != MAGIC_NUMBER) {
                throw new IOException("Corrupt data: Unexpected magic number.");
            }

            byte[] targetFilePathBytes = new byte[bb.getInt()];
            bb.get(targetFilePathBytes);
            String targetFilePath = new String(targetFilePathBytes, CHARSET);

            byte[] tempFilePathBytes = new byte[bb.getInt()];
            bb.get(tempFilePathBytes);
            String tempFilePath = new String(tempFilePathBytes, CHARSET);

            return new HadoopPathBasedPendingFileRecoverable(
                    new Path(targetFilePath), new Path(tempFilePath));
        }

        private HadoopPathBasedPendingFileRecoverable deserializeV2(byte[] serialized)
                throws IOException {
            final ByteBuffer bb = ByteBuffer.wrap(serialized).order(ByteOrder.LITTLE_ENDIAN);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Address the cause reported by the error message: Corrupt data: Unexpected magic number.
2. Verify the inputs, configuration values, and classpath/dependency setup related to this operation, then retry.

Example fix

Correct the condition described ("Corrupt data: Unexpected magic number.") and rerun the job or command.

When it happens

Trigger: Triggered at runtime when the operation fails because: Corrupt data: Unexpected magic number.

Common situations: Commonly caused by misconfiguration, missing dependencies or files, unsupported types or operations, or invalid user input leading to: Corrupt data: Unexpected magic number.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/fafe47f8ecc82384. Report an issue: GitHub.