apache/kafka · error · IOException

Stream ended prematurely

Error message

Stream ended prematurely

What it means

Thrown in Lz4BlockInputStream.readHeader() when the input ByteBuffer has fewer than 6 bytes remaining at the start of a frame, meaning it cannot even contain the 4-byte magic plus FLG/BD descriptor bytes of an LZ4 frame. It signals a truncated/corrupt LZ4 frame header and is raised as IOException(PREMATURE_EOS). This is the earliest possible failure when decompressing a record batch whose LZ4 payload was cut off.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java:118

    /**
     * Check whether KafkaLZ4BlockInputStream is configured to ignore the
     * Frame Descriptor checksum, which is useful for compatibility with
     * old client implementations that use incorrect checksum calculations.
     */
    public boolean ignoreFlagDescriptorChecksum() {
        return this.ignoreFlagDescriptorChecksum;
    }

    /**
     * Reads the magic number and frame descriptor from input buffer.
     *
     * @throws IOException
     */
    private void readHeader() throws IOException {
        // read first 6 bytes into buffer to check magic and FLG/BD descriptor flags
        if (in.remaining() < 6) {
            throw new IOException(PREMATURE_EOS);
        }

        if (MAGIC != in.getInt()) {
            throw new IOException(NOT_SUPPORTED);
        }
        // mark start of data to checksum
        in.mark();

        flg = FLG.fromByte(in.get());
        maxBlockSize = BD.fromByte(in.get()).getBlockMaximumSize();

        if (flg.isContentSizeSet()) {
            if (in.remaining() < 8) {
                throw new IOException(PREMATURE_EOS);
            }
            in.position(in.position() + 8);
        }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Verify the source ByteBuffer contains a complete LZ4 frame before decompression (at minimum the header plus one block and end mark).
2. Check upstream transport/storage for truncation: network/replication errors, partial writes, or filesystem corruption.
3. If you sliced the buffer manually, ensure position/limit correctly bound the full compressed payload and weren't advanced past the start.
4. Reproduce with a known-good batch to rule out producer-side corruption.

// Header read needs at least 6 bytes; truncated stream surfaces as IOException
try {
    try (Lz4BlockInputStream in = new Lz4BlockInputStream(buffer, ignoreFlagDescriptorChecksum)) {
        // ... read fully ...
    }
} catch (IOException e) {
    // "Stream ended prematurely" -- source buffer is shorter than the LZ4 frame header;
    // record offset, skip the record, do not retry the same payload
}

When it happens

Trigger: Constructing/reading an Lz4BlockInputStream over a ByteBuffer with <6 bytes; happens during Kafka record-batch decompression when a compressed record batch is truncated by a network error, partial file, log corruption, or when the buffer passed in only contains a fragment of the LZ4 frame.

Common situations: Truncated log segment files after a broker crash or disk fault; partial messages from a producer/consumer over an unstable connection; mis-split buffers before passing them to the decompressor; custom serdes that slice a record batch incorrectly.

Related errors

• Stream frame descriptor corrupted
• Block size %d exceeded max: %d
• Stream unsupported (invalid magic bytes)
• mark not supported
• reset not supported

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/5ee846396695a03c.json. Report an issue: GitHub.