apache/kafka · error · KafkaException

Error checking for remaining bytes after reading batch

Error message

Error checking for remaining bytes after reading batch

What it means

Thrown by StreamRecordIterator.ensureNoneRemaining() when, after reading the declared record count, it calls inputStream.read() to verify end-of-stream and that single read throws IOException. This is the streaming-iterator's tail integrity check failing not because bytes remain (that is error 520) but because the underlying decompressed stream cannot be probed for EOF. Wrapped as KafkaException because it is an I/O failure in the decompressor, not a structural record-count violation.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java:651

        abstract Record doReadRecord(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) throws IOException;

        @Override
        protected Record readNext(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) {
            try {
                return doReadRecord(baseOffset, baseTimestamp, baseSequence, logAppendTime);
            } catch (IllegalArgumentException e) {
                throw new InvalidRecordException("Incorrect declared batch size, premature EOF reached", e);
            } catch (IOException e) {
                throw new KafkaException("Failed to decompress record stream", e);
            }
        }

        @Override
        protected boolean ensureNoneRemaining() {
            try {
                return inputStream.read() == -1;
            } catch (IOException e) {
                throw new KafkaException("Error checking for remaining bytes after reading batch", e);
            }
        }

        @Override
        public void close() {
            try {
                inputStream.close();
            } catch (IOException e) {
                throw new KafkaException("Failed to close record stream", e);
            }
        }
    }

    static class DefaultFileChannelRecordBatch extends FileLogInputStream.FileChannelRecordBatch {

        DefaultFileChannelRecordBatch(long offset,
                                      byte magic,
                                      FileRecords fileRecords,

View on GitHub (pinned to c31c9215e1)

Solutions

Dump the segment with kafka-dump-log.sh --deep-iteration to confirm the batch decompresses partially then fails on the trailing read.
Recover a clean copy from an in-sync replica via preferred leader election or reassignment; for RF=1 topics truncate the corrupt tail.
If reproducible with a specific producer, capture the exact batch bytes and file a test against the codec; align compression library versions between producer and broker.
Check dmesg / SMART for disk errors if corruption is hardware-origin.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  while (recordIterator.hasNext()) { Record r = recordIterator.next(); /* ... */ }
} catch (org.apache.kafka.common.KafkaException e) {
  // ensureNoneRemaining() hit IOException on the underlying stream — I/O fault, not corruption.
  log.warn("I/O error validating batch tail: {}", e.getCause());
}

Prevention

Keep the underlying FileChannel / InputStream open for the full lifetime of the iterator; closing it mid-iteration triggers this IOException.
Do not share one FileRecords instance across concurrent iterators — close one and the others see read failures.
Treat as transient I/O: re-open the segment/channel and retry iteration from the batch start position before giving up.
Check that the segment file has not been deleted/compacted out from under the reader (a common cause during log cleanup).
Wrap iterator use in try-with-resources over the CloseableIterator and re-derive FileRecords in the recovery path.

When it happens

Trigger: Iteration through a compressed batch's streamingIterator where, after the last declared record, inputStream.read() raises IOException (the decompressor's read method fails). The path is next() -> readRecords==numRecords -> ensureNoneRemaining() -> inputStream.read() throws.

Common situations: A decompressor (gzip Inflater, snappy, lz4) that fails on a final read because the trailing compressed bytes are truncated or checksum-invalid even though enough records were decodable. Seen with truncated segments, partially flushed batches, or after a forced kill during a compressed append. Disk/page-cache bit rot is another source.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/73fe4be1019c518b.json. Report an issue: GitHub.