{"id":"00921e8686e0a594","repo":"apache/kafka","slug":"stream-unsupported-invalid-magic-bytes","errorCode":null,"errorMessage":"Stream unsupported (invalid magic bytes)","messagePattern":"Stream unsupported \\(invalid magic bytes\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java","lineNumber":122,"sourceCode":"     * old client implementations that use incorrect checksum calculations.\n     */\n    public boolean ignoreFlagDescriptorChecksum() {\n        return this.ignoreFlagDescriptorChecksum;\n    }\n\n    /**\n     * Reads the magic number and frame descriptor from input buffer.\n     *\n     * @throws IOException\n     */\n    private void readHeader() throws IOException {\n        // read first 6 bytes into buffer to check magic and FLG/BD descriptor flags\n        if (in.remaining() < 6) {\n            throw new IOException(PREMATURE_EOS);\n        }\n\n        if (MAGIC != in.getInt()) {\n            throw new IOException(NOT_SUPPORTED);\n        }\n        // mark start of data to checksum\n        in.mark();\n\n        flg = FLG.fromByte(in.get());\n        maxBlockSize = BD.fromByte(in.get()).getBlockMaximumSize();\n\n        if (flg.isContentSizeSet()) {\n            if (in.remaining() < 8) {\n                throw new IOException(PREMATURE_EOS);\n            }\n            in.position(in.position() + 8);\n        }\n\n        // Final byte of Frame Descriptor is HC checksum\n\n        // Old implementations produced incorrect HC checksums\n        if (ignoreFlagDescriptorChecksum) {","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java#L104-L140","documentation":"Thrown in readHeader() when the first 4 bytes read from the LZ4 frame do not equal Lz4BlockOutputStream.MAGIC. Kafka writes an LZ4 frame with a specific magic number; a mismatch means the buffer is not a valid Kafka LZ4 frame (e.g. it is a raw LZ4 block, a different format like gzip/snappy/zstd, or random data). Raised as IOException(NOT_SUPPORTED) to distinguish 'wrong format' from 'truncated'.","triggerScenarios":"Decompressing a ByteBuffer whose payload was produced by a non-frame LZ4 compressor (raw LZ4 block), by a different codec, or by mismatched producer/broker compression settings (e.g. producer sends snappy/zstd but consumer expects LZ4). Also when buffer position is wrong, so the magic bytes are read from the middle of the payload.","commonSituations":"Compression-type mismatch between producer (compression.type) and consumer/decompressor; using a generic LZ4 library that emits the LZ4 block format instead of the LZ4 frame format Kafka expects; buffer offset/position bugs causing the magic read to land on data; corrupted records from a buggy custom serializer.","solutions":["Confirm producer and consumer use the same compression.type (LZ4) on both sides.","Ensure the LZ4 payload is the LZ4 Frame format Kafka uses, not raw LZ4 blocks produced by other libraries.","Check that the ByteBuffer passed to the decompressor is positioned at the very start of the compressed frame and not advanced/offset.","If integrating non-Kafka data, re-encode it using Kafka's Lz4BlockOutputStream first."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Best-effort pre-check: LZ4 Kafka frame magic is 0x4D415A4B (little-endian 'KAFM')\nByteBuffer probe = buffer.duplicate();\nif (probe.remaining() >= 4 && probe.getInt() != Lz4BlockInputStream.MAGIC) {\n    // not an LZ4 Kafka block; pick a different Compression / reject record\n}","typeGuard":null,"tryCatchPattern":"try {\n    try (Lz4BlockInputStream in = new Lz4BlockInputStream(buffer, ignoreFlagDescriptorChecksum)) {\n        // ... read ...\n    }\n} catch (IOException e) {\n    // \"Stream unsupported (invalid magic bytes)\" -- payload is not LZ4 or uses a non-Kafka LZ4 frame;\n    // do not retry; verify producer/broker compression codec matches consumer expectations\n}","preventionTips":["Ensure producer and consumer agree on compression.type (e.g. 'lz4'); mismatched codecs produce magic-byte failures.","When compression is negotiated per-topic, validate the configured codec against what the broker advertises.","Do not feed arbitrary/uncompressed bytes into Lz4BlockInputStream; it expects the Kafka LZ4 frame format.","Distinguish this from generic truncation: magic-byte mismatch means wrong data, not short data."],"tags":["compression","lz4","format-mismatch","io","java"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}