apache/kafka · error · KafkaException

Failed to decompress record stream

Error message

Failed to decompress record stream

What it means

Thrown by StreamRecordIterator.readNext() when doReadRecord() throws an IOException during decompression of the record stream inside a v2 batch. Unlike the premature-EOF case (which is an IllegalArgumentException wrapped as InvalidRecordException), a raw IOException from the decompressor is wrapped as a KafkaException because the cause is an I/O / codec-layer failure rather than a count mismatch. This indicates the compressed payload itself is unreadable by the configured CompressionType.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java:642

    // visible for testing
    abstract class StreamRecordIterator extends RecordIterator {
        private final InputStream inputStream;

        StreamRecordIterator(InputStream inputStream) {
            super();
            this.inputStream = inputStream;
        }

        abstract Record doReadRecord(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) throws IOException;

        @Override
        protected Record readNext(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) {
            try {
                return doReadRecord(baseOffset, baseTimestamp, baseSequence, logAppendTime);
            } catch (IllegalArgumentException e) {
                throw new InvalidRecordException("Incorrect declared batch size, premature EOF reached", e);
            } catch (IOException e) {
                throw new KafkaException("Failed to decompress record stream", e);
            }
        }

        @Override
        protected boolean ensureNoneRemaining() {
            try {
                return inputStream.read() == -1;
            } catch (IOException e) {
                throw new KafkaException("Error checking for remaining bytes after reading batch", e);
            }
        }

        @Override
        public void close() {
            try {
                inputStream.close();
            } catch (IOException e) {
                throw new KafkaException("Failed to close record stream", e);

View on GitHub (pinned to c31c9215e1)

Solutions

1. Confirm the batch's CompressionType (from attributes byte) against what the producer actually used; mismatched codec framing is the most common non-corruption cause.
2. Dump the batch with kafka-dump-log.sh --deep-iteration to see the codec and the offset where decompression fails.
3. If a codec/library version skew between producer and broker, align the compression library versions (or switch compression.type to a more portable codec like lz4 framed) and re-produce.
4. If genuine corruption, recover from an ISR replica or truncate the corrupt tail of the segment.

Example fix

// before: producer uses gzip, broker JVM has a broken zlib/native lib
props.put("compression.type", "gzip");
// after: use lz4 which is framed and version-stable across Kafka builds
props.put("compression.type", "lz4");

try {
  CloseableIterator<Record> it = batch.streamingIterator(bufferSupplier);
  try { while (it.hasNext()) { Record r = it.next(); /* ... */ } }
  finally { it.close(); }
} catch (org.apache.kafka.common.KafkaException e) {
  if (e.getCause() instanceof java.io.IOException) {
    // Decompression I/O failure — usually codec mismatch or corruption inside compressed payload.
    log.warn("Decompression failed for batch at {}: {}", batch.baseOffset(), e.getCause().getMessage());
  } else { throw e; }
}

When it happens

Trigger: Calling streamingIterator on a compressed DefaultRecordBatch where the inflater/decompressor for the batch's CompressionType (gzip/snappy/lz4/zstd) throws IOException while decoding the per-record bytes inside the batch payload. Triggered during consumer fetch handling, log validation on append, or any path that materializes records from a streaming batch.

Common situations: Codec implementation mismatch (e.g. snappy vs lz4 framing variants), a truncated or bit-flipped compressed payload from disk corruption, an OS-level FS or page-cache issue, a producer using a compression library version that emits frames the broker's codec rejects, or reading data written by a non-standard client that mislabels the compression type in the batch attributes.

Related errors

• Error checking for remaining bytes after reading batch
• Failed to close record stream
• Failed to load record batch at position {} from {}
• Incorrect declared batch size, records still remaining in fi
• Found record size %d smaller than minimum record overhead (%

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/c0a6590052b3950a.json. Report an issue: GitHub.