apache/kafka · error · KafkaException

Failed to close record stream

Error message

Failed to close record stream

What it means

Thrown by StreamRecordIterator.close() when closing the underlying InputStream (the decompressed record stream) raises IOException. The iterator implements CloseableIterator and wraps the close failure as KafkaException because, by this point, the records have already been read; the failure is purely a resource-release problem in the decompressor or its wrapper stream.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java:660

                throw new KafkaException("Failed to decompress record stream", e);
            }
        }

        @Override
        protected boolean ensureNoneRemaining() {
            try {
                return inputStream.read() == -1;
            } catch (IOException e) {
                throw new KafkaException("Error checking for remaining bytes after reading batch", e);
            }
        }

        @Override
        public void close() {
            try {
                inputStream.close();
            } catch (IOException e) {
                throw new KafkaException("Failed to close record stream", e);
            }
        }
    }

    static class DefaultFileChannelRecordBatch extends FileLogInputStream.FileChannelRecordBatch {

        DefaultFileChannelRecordBatch(long offset,
                                      byte magic,
                                      FileRecords fileRecords,
                                      int position,
                                      int batchSize) {
            super(offset, magic, fileRecords, position, batchSize);
        }

        @Override
        protected RecordBatch toMemoryRecordBatch(ByteBuffer buffer) {
            return new DefaultRecordBatch(buffer);
        }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Look at the preceding log lines / suppressed exceptions: this is usually secondary to an earlier IOException from readNext; fix the root cause (corruption or codec mismatch) first.
2. Ensure callers close iterators in finally / try-with-resources so a failed read does not leak the stream before close is attempted.
3. If using a custom record interceptor or InputStream wrapper, verify its close() is exception-safe and does not mask the real failure.
4. Recover the corrupt segment from ISR or truncate as with other corruption errors.

Example fix

// before: iterator leaked on read failure, close then throws later
Iterator<Record> it = batch.streamingIterator(supplier);
Record r = it.next(); // throws
// it never closed
// after: try-with-resources guarantees close and surfaces the first failure
try (CloseableIterator<Record> it = batch.streamingIterator(supplier)) {
    while (it.hasNext()) consume(it.next());
}

CloseableIterator<Record> it = batch.streamingIterator(bufferSupplier);
try {
  while (it.hasNext()) { /* process it.next() */ }
} finally {
  try { it.close(); }
  catch (org.apache.kafka.common.KafkaException closeErr) {
    // Best-effort close: data already consumed; log and suppress, never re-throw over a primary result.
    log.debug("Suppressed stream close failure", closeErr);
  }
}

When it happens

Trigger: Any code path that calls .close() on a streamingIterator obtained from a compressed DefaultRecordBatch (consumer fetch release, log cleaner, replication fetch) where the underlying decompressor's close() throws IOException. Try-with-resources over the iterator or explicit close in a finally block triggers it.

Common situations: Decompressor (Inflater/zip) close() failing due to an already-failed decompression state, a custom InputStream wrapper throwing on close, or an underlying channel/FS error during resource release. Often appears as a secondary exception after an earlier decompression failure. Rare on its own; usually indicates the stream was already in a bad state.

Related errors

• Failed to decompress record stream
• Error checking for remaining bytes after reading batch
• Incorrect declared batch size, records still remaining in fi
• Found record size %d smaller than minimum record overhead (%
• Failed to load record batch at position {} from {}

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/a71ff3ceae50f16a.json. Report an issue: GitHub.