apache/kafka · error · IOException

Stream unsupported (invalid magic bytes)

Error message

Stream unsupported (invalid magic bytes)

What it means

Thrown in readHeader() when the first 4 bytes read from the LZ4 frame do not equal Lz4BlockOutputStream.MAGIC. Kafka writes an LZ4 frame with a specific magic number; a mismatch means the buffer is not a valid Kafka LZ4 frame (e.g. it is a raw LZ4 block, a different format like gzip/snappy/zstd, or random data). Raised as IOException(NOT_SUPPORTED) to distinguish 'wrong format' from 'truncated'.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java:122

     * old client implementations that use incorrect checksum calculations.
     */
    public boolean ignoreFlagDescriptorChecksum() {
        return this.ignoreFlagDescriptorChecksum;
    }

    /**
     * Reads the magic number and frame descriptor from input buffer.
     *
     * @throws IOException
     */
    private void readHeader() throws IOException {
        // read first 6 bytes into buffer to check magic and FLG/BD descriptor flags
        if (in.remaining() < 6) {
            throw new IOException(PREMATURE_EOS);
        }

        if (MAGIC != in.getInt()) {
            throw new IOException(NOT_SUPPORTED);
        }
        // mark start of data to checksum
        in.mark();

        flg = FLG.fromByte(in.get());
        maxBlockSize = BD.fromByte(in.get()).getBlockMaximumSize();

        if (flg.isContentSizeSet()) {
            if (in.remaining() < 8) {
                throw new IOException(PREMATURE_EOS);
            }
            in.position(in.position() + 8);
        }

        // Final byte of Frame Descriptor is HC checksum

        // Old implementations produced incorrect HC checksums
        if (ignoreFlagDescriptorChecksum) {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Confirm producer and consumer use the same compression.type (LZ4) on both sides.
2. Ensure the LZ4 payload is the LZ4 Frame format Kafka uses, not raw LZ4 blocks produced by other libraries.
3. Check that the ByteBuffer passed to the decompressor is positioned at the very start of the compressed frame and not advanced/offset.
4. If integrating non-Kafka data, re-encode it using Kafka's Lz4BlockOutputStream first.

// Best-effort pre-check: LZ4 Kafka frame magic is 0x4D415A4B (little-endian 'KAFM')
ByteBuffer probe = buffer.duplicate();
if (probe.remaining() >= 4 && probe.getInt() != Lz4BlockInputStream.MAGIC) {
    // not an LZ4 Kafka block; pick a different Compression / reject record
}

try {
    try (Lz4BlockInputStream in = new Lz4BlockInputStream(buffer, ignoreFlagDescriptorChecksum)) {
        // ... read ...
    }
} catch (IOException e) {
    // "Stream unsupported (invalid magic bytes)" -- payload is not LZ4 or uses a non-Kafka LZ4 frame;
    // do not retry; verify producer/broker compression codec matches consumer expectations
}

When it happens

Trigger: Decompressing a ByteBuffer whose payload was produced by a non-frame LZ4 compressor (raw LZ4 block), by a different codec, or by mismatched producer/broker compression settings (e.g. producer sends snappy/zstd but consumer expects LZ4). Also when buffer position is wrong, so the magic bytes are read from the middle of the payload.

Common situations: Compression-type mismatch between producer (compression.type) and consumer/decompressor; using a generic LZ4 library that emits the LZ4 block format instead of the LZ4 frame format Kafka expects; buffer offset/position bugs causing the magic read to land on data; corrupted records from a buggy custom serializer.

Related errors

• Stream ended prematurely
• Stream frame descriptor corrupted
• Block size %d exceeded max: %d
• mark not supported
• reset not supported

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/00921e8686e0a594.json. Report an issue: GitHub.