What causes "Stream frame descriptor corrupted" in apache/kafka?

If reading data written by old Kafka clients, construct Lz4BlockInputStream with ignoreFlagDescriptorChecksum=true (the path Kafka uses for legacy batches) to bypass the HC check.

apache/kafka · error · IOException

Stream frame descriptor corrupted

Error message

Stream frame descriptor corrupted

What it means

Thrown in readHeader() when the HC (Header Checksum) byte computed over the FLG/BD descriptor (and optional content size) does not equal the byte stored in the frame. LZ4 frames include a one-byte XXHash32-derived checksum of the frame descriptor; a mismatch means the descriptor bytes are corrupt or were produced by an implementation with an incorrect checksum calculation. Raised as IOException(DESCRIPTOR_HASH_MISMATCH) unless ignoreFlagDescriptorChecksum was set to bypass it.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java:150

                throw new IOException(PREMATURE_EOS);
            }
            in.position(in.position() + 8);
        }

        // Final byte of Frame Descriptor is HC checksum

        // Old implementations produced incorrect HC checksums
        if (ignoreFlagDescriptorChecksum) {
            in.position(in.position() + 1);
            return;
        }

        int len = in.position() - in.reset().position();

        int hash = CHECKSUM.hash(in, in.position(), len, 0);
        in.position(in.position() + len);
        if (in.get() != (byte) ((hash >> 8) & 0xFF)) {
            throw new IOException(DESCRIPTOR_HASH_MISMATCH);
        }
    }

    /**
     * Decompresses (if necessary) buffered data, optionally computes and validates a XXHash32 checksum, and writes the
     * result to a buffer.
     *
     * @throws IOException
     */
    private void readBlock() throws IOException {
        if (in.remaining() < 4) {
            throw new IOException(PREMATURE_EOS);
        }

        int blockSize = in.getInt();
        boolean compressed = (blockSize & LZ4_FRAME_INCOMPRESSIBLE_MASK) == 0;
        blockSize &= ~LZ4_FRAME_INCOMPRESSIBLE_MASK;

View on GitHub (pinned to c31c9215e1)

Solutions

If reading data written by old Kafka clients, construct Lz4BlockInputStream with ignoreFlagDescriptorChecksum=true (the path Kafka uses for legacy batches) to bypass the HC check.
If data is from a conformant producer, investigate corruption: disk, network, buffer reuse, or concurrent modification.
Re-encode the data with a current Kafka client using Lz4BlockOutputStream to get a correct descriptor checksum.
Confirm the LZ4 encoder (if non-Kafka) implements the v1.5.1 frame HC byte correctly.

Defensive patterns

Strategy: try-catch

Validate before calling

// Optional: verify descriptor checksum yourself only if interoperating with hand-rolled LZ4 producers.
// Normally, do NOT replicate the XXHash32 check; rely on the stream to throw.
// If interoperating with legacy producers known to emit bad HC, construct with ignoreFlagDescriptorChecksum=true:
new Lz4BlockInputStream(buffer, /* ignoreFlagDescriptorChecksum */ true);

Try / catch

try {
    try (Lz4BlockInputStream in = new Lz4BlockInputStream(buffer, ignoreFlagDescriptorChecksum)) {
        // ... read ...
    }
} catch (IOException e) {
    // "Stream frame descriptor corrupted" -- HC checksum mismatch;
    // do NOT silently set ignoreFlagDescriptorChecksum=true to suppress; investigate the producer
}

Prevention

Do NOT default ignoreFlagDescriptorChecksum=true to mask the failure; it exists only for legacy producer compatibility.
A genuine HC mismatch usually means bit-flips or a non-conformant producer; investigate root cause.
Pin producer and consumer LZ4 implementations to compatible versions; mismatched XXHash32 variants cause spurious failures.
Capture the offending bytes for offline lz4 --decompress reproduction when diagnosing.

When it happens

Trigger: Decompressing an LZ4 frame whose descriptor bytes were corrupted, truncated-and-rewritten, or produced by old/buggy Kafka clients (<0.10) that computed the HC byte incorrectly, while the consumer was constructed with ignoreFlagDescriptorChecksum=false. Also triggered by non-conforming third-party LZ4 encoders.

Common situations: Interoperating with very old Kafka clients whose LZ4 HC calculation was wrong; bit-rot/corruption on disk; memory/network bit flips; hand-edited frames; consumer constructed without the legacy-compat flag when reading legacy data.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/167b8cf040a62537.json. Report an issue: GitHub.