{"record":{"id":"1fb5a7936049b92c","repo":"apache/flink","slug":"corrupt-data-unexpected-magic-number","errorCode":null,"errorMessage":"Corrupt data: Unexpected magic number.","messagePattern":"Corrupt data: Unexpected magic number\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableSerializer.java","lineNumber":80,"sourceCode":"\n        return targetBytes;\n    }\n\n    @Override\n    public LocalRecoverable deserialize(int version, byte[] serialized) throws IOException {\n        switch (version) {\n            case 1:\n                return deserializeV1(serialized);\n            default:\n                throw new IOException(\"Unrecognized version or corrupt state: \" + version);\n        }\n    }\n\n    private static LocalRecoverable deserializeV1(byte[] serialized) throws IOException {\n        final ByteBuffer bb = ByteBuffer.wrap(serialized).order(ByteOrder.LITTLE_ENDIAN);\n\n        if (bb.getInt() != MAGIC_NUMBER) {\n            throw new IOException(\"Corrupt data: Unexpected magic number.\");\n        }\n\n        final long offset = bb.getLong();\n        final byte[] targetFileBytes = new byte[bb.getInt()];\n        final byte[] tempFileBytes = new byte[bb.getInt()];\n        bb.get(targetFileBytes);\n        bb.get(tempFileBytes);\n\n        final String targetPath = new String(targetFileBytes, CHARSET);\n        final String tempPath = new String(tempFileBytes, CHARSET);\n\n        return new LocalRecoverable(new File(targetPath), new File(tempPath), offset);\n    }\n}\n","sourceCodeStart":62,"sourceCodeEnd":95,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableSerializer.java#L62-L95","documentation":"Thrown inside LocalRecoverableSerializer.deserializeV1 when the first four bytes of the deserialized payload (read in LITTLE_ENDIAN) do not equal the expected MAGIC_NUMBER constant (0x1e744b57). The magic number is a sanity sentinel written at serialize time to confirm the byte stream is genuinely a LocalRecoverable payload and not random or mis-framed data.","triggerScenarios":"Calling LocalRecoverableSerializer.deserialize(1, bytes) where bytes were not produced by the matching serialize() method; passing a byte array whose first four bytes were truncated, shifted, or belong to a different serializer's format; accidentally feeding the full SimpleVersionedSerialization framing (version+length header) directly to deserialize instead of stripping the 8-byte header first.","commonSituations":"Checkpoint/savepoint corruption on local disk or in-memory transport; manual byte-array manipulation or incorrect offsets when slicing serialized data; version field says 1 but the payload came from a different recoverable type (e.g. HadoopRecoverable).","solutions":["Ensure the byte array passed to deserialize is exactly the output of LocalRecoverableSerializer.serialize — no extra framing bytes prepended or appended.","If reading from a SimpleVersionedSerialization stream, use readVersionAndDeSerialize which strips the version+length header automatically, rather than calling deserialize directly.","Verify the checkpoint file is not corrupted by checking its size against expected metadata.","If the data is genuinely from a different source, use the correct serializer for that recoverable type."],"exampleFix":"// before — raw bytes fed directly, header not stripped\nbyte[] raw = Files.readAllBytes(metaPath);\nLocalRecoverable r = LocalRecoverableSerializer.INSTANCE.deserialize(1, raw);\n\n// after — use the standard framing utility\ntry (DataInputViewStreamWrapper in = new DataInputViewStreamWrapper(Files.newInputStream(metaPath))) {\n    LocalRecoverable r = SimpleVersionedSerialization.readVersionAndDeSerialize(\n        LocalRecoverableSerializer.INSTANCE, in);\n}","handlingStrategy":"try-catch","validationCode":"ByteBuffer probe = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN);\nif (probe.remaining() < 4 || probe.getInt() != 0x1e744b57) {\n    throw new IOException(\"Not a valid LocalRecoverable payload (bad magic number)\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    LocalRecoverable r = LocalRecoverableSerializer.INSTANCE.deserialize(1, bytes);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"magic number\")) {\n        // payload is not LocalRecoverable data; use correct serializer or regenerate\n    }\n    throw e;\n}","preventionTips":["Use SimpleVersionedSerialization framing consistently so bytes are never fed raw to deserialize.","Never manually slice or prepend bytes to a serialized LocalRecoverable payload.","Store recovery metadata with its full framing to avoid offset errors."],"tags":["serialization","local-filesystem","data-corruption","magic-number"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}