{"id":"c0a6590052b3950a","repo":"apache/kafka","slug":"failed-to-decompress-record-stream","errorCode":null,"errorMessage":"Failed to decompress record stream","messagePattern":"Failed to decompress record stream","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java","lineNumber":642,"sourceCode":"    // visible for testing\n    abstract class StreamRecordIterator extends RecordIterator {\n        private final InputStream inputStream;\n\n        StreamRecordIterator(InputStream inputStream) {\n            super();\n            this.inputStream = inputStream;\n        }\n\n        abstract Record doReadRecord(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) throws IOException;\n\n        @Override\n        protected Record readNext(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) {\n            try {\n                return doReadRecord(baseOffset, baseTimestamp, baseSequence, logAppendTime);\n            } catch (IllegalArgumentException e) {\n                throw new InvalidRecordException(\"Incorrect declared batch size, premature EOF reached\", e);\n            } catch (IOException e) {\n                throw new KafkaException(\"Failed to decompress record stream\", e);\n            }\n        }\n\n        @Override\n        protected boolean ensureNoneRemaining() {\n            try {\n                return inputStream.read() == -1;\n            } catch (IOException e) {\n                throw new KafkaException(\"Error checking for remaining bytes after reading batch\", e);\n            }\n        }\n\n        @Override\n        public void close() {\n            try {\n                inputStream.close();\n            } catch (IOException e) {\n                throw new KafkaException(\"Failed to close record stream\", e);","sourceCodeStart":624,"sourceCodeEnd":660,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java#L624-L660","documentation":"Thrown by StreamRecordIterator.readNext() when doReadRecord() throws an IOException during decompression of the record stream inside a v2 batch. Unlike the premature-EOF case (which is an IllegalArgumentException wrapped as InvalidRecordException), a raw IOException from the decompressor is wrapped as a KafkaException because the cause is an I/O / codec-layer failure rather than a count mismatch. This indicates the compressed payload itself is unreadable by the configured CompressionType.","triggerScenarios":"Calling streamingIterator on a compressed DefaultRecordBatch where the inflater/decompressor for the batch's CompressionType (gzip/snappy/lz4/zstd) throws IOException while decoding the per-record bytes inside the batch payload. Triggered during consumer fetch handling, log validation on append, or any path that materializes records from a streaming batch.","commonSituations":"Codec implementation mismatch (e.g. snappy vs lz4 framing variants), a truncated or bit-flipped compressed payload from disk corruption, an OS-level FS or page-cache issue, a producer using a compression library version that emits frames the broker's codec rejects, or reading data written by a non-standard client that mislabels the compression type in the batch attributes.","solutions":["Confirm the batch's CompressionType (from attributes byte) against what the producer actually used; mismatched codec framing is the most common non-corruption cause.","Dump the batch with kafka-dump-log.sh --deep-iteration to see the codec and the offset where decompression fails.","If a codec/library version skew between producer and broker, align the compression library versions (or switch compression.type to a more portable codec like lz4 framed) and re-produce.","If genuine corruption, recover from an ISR replica or truncate the corrupt tail of the segment."],"exampleFix":"// before: producer uses gzip, broker JVM has a broken zlib/native lib\nprops.put(\"compression.type\", \"gzip\");\n// after: use lz4 which is framed and version-stable across Kafka builds\nprops.put(\"compression.type\", \"lz4\");","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  CloseableIterator<Record> it = batch.streamingIterator(bufferSupplier);\n  try { while (it.hasNext()) { Record r = it.next(); /* ... */ } }\n  finally { it.close(); }\n} catch (org.apache.kafka.common.KafkaException e) {\n  if (e.getCause() instanceof java.io.IOException) {\n    // Decompression I/O failure — usually codec mismatch or corruption inside compressed payload.\n    log.warn(\"Decompression failed for batch at {}: {}\", batch.baseOffset(), e.getCause().getMessage());\n  } else { throw e; }\n}","preventionTips":["Keep compression.type consistent between producer and consumer; an unknown/unavailable codec on the classpath surfaces as a decompression IOException.","Ensure the compression library (e.g., lz4-java, snappy-java, zstd-jni) is on the consumer classpath at a compatible version.","Do not retry the same compressed batch on failure — corrupt payloads will not decompress on re-read; skip the batch.","Validate batches on the broker (compression validation is part of log append validation); a corrupt compressed batch reaching a consumer means broker validation was bypassed or the file was damaged post-append.","When implementing a custom LogInputStream, never hand a truncated ByteBuffer to a streaming iterator — size the buffer to the declared batch length first."],"tags":["kafka","records","compression","io-exception","codec","broker"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}