{"id":"73fe4be1019c518b","repo":"apache/kafka","slug":"error-checking-for-remaining-bytes-after-reading-b","errorCode":null,"errorMessage":"Error checking for remaining bytes after reading batch","messagePattern":"Error checking for remaining bytes after reading batch","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java","lineNumber":651,"sourceCode":"        abstract Record doReadRecord(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) throws IOException;\n\n        @Override\n        protected Record readNext(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) {\n            try {\n                return doReadRecord(baseOffset, baseTimestamp, baseSequence, logAppendTime);\n            } catch (IllegalArgumentException e) {\n                throw new InvalidRecordException(\"Incorrect declared batch size, premature EOF reached\", e);\n            } catch (IOException e) {\n                throw new KafkaException(\"Failed to decompress record stream\", e);\n            }\n        }\n\n        @Override\n        protected boolean ensureNoneRemaining() {\n            try {\n                return inputStream.read() == -1;\n            } catch (IOException e) {\n                throw new KafkaException(\"Error checking for remaining bytes after reading batch\", e);\n            }\n        }\n\n        @Override\n        public void close() {\n            try {\n                inputStream.close();\n            } catch (IOException e) {\n                throw new KafkaException(\"Failed to close record stream\", e);\n            }\n        }\n    }\n\n    static class DefaultFileChannelRecordBatch extends FileLogInputStream.FileChannelRecordBatch {\n\n        DefaultFileChannelRecordBatch(long offset,\n                                      byte magic,\n                                      FileRecords fileRecords,","sourceCodeStart":633,"sourceCodeEnd":669,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java#L633-L669","documentation":"Thrown by StreamRecordIterator.ensureNoneRemaining() when, after reading the declared record count, it calls inputStream.read() to verify end-of-stream and that single read throws IOException. This is the streaming-iterator's tail integrity check failing not because bytes remain (that is error 520) but because the underlying decompressed stream cannot be probed for EOF. Wrapped as KafkaException because it is an I/O failure in the decompressor, not a structural record-count violation.","triggerScenarios":"Iteration through a compressed batch's streamingIterator where, after the last declared record, inputStream.read() raises IOException (the decompressor's read method fails). The path is next() -> readRecords==numRecords -> ensureNoneRemaining() -> inputStream.read() throws.","commonSituations":"A decompressor (gzip Inflater, snappy, lz4) that fails on a final read because the trailing compressed bytes are truncated or checksum-invalid even though enough records were decodable. Seen with truncated segments, partially flushed batches, or after a forced kill during a compressed append. Disk/page-cache bit rot is another source.","solutions":["Dump the segment with kafka-dump-log.sh --deep-iteration to confirm the batch decompresses partially then fails on the trailing read.","Recover a clean copy from an in-sync replica via preferred leader election or reassignment; for RF=1 topics truncate the corrupt tail.","If reproducible with a specific producer, capture the exact batch bytes and file a test against the codec; align compression library versions between producer and broker.","Check dmesg / SMART for disk errors if corruption is hardware-origin."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  while (recordIterator.hasNext()) { Record r = recordIterator.next(); /* ... */ }\n} catch (org.apache.kafka.common.KafkaException e) {\n  // ensureNoneRemaining() hit IOException on the underlying stream — I/O fault, not corruption.\n  log.warn(\"I/O error validating batch tail: {}\", e.getCause());\n}","preventionTips":["Keep the underlying FileChannel / InputStream open for the full lifetime of the iterator; closing it mid-iteration triggers this IOException.","Do not share one FileRecords instance across concurrent iterators — close one and the others see read failures.","Treat as transient I/O: re-open the segment/channel and retry iteration from the batch start position before giving up.","Check that the segment file has not been deleted/compacted out from under the reader (a common cause during log cleanup).","Wrap iterator use in try-with-resources over the CloseableIterator and re-derive FileRecords in the recovery path."],"tags":["kafka","records","compression","io-exception","streaming-iterator","broker"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}