{"id":"a9d3ea55bceaea21","repo":"apache/kafka","slug":"incorrect-declared-batch-size-premature-eof-reach","errorCode":null,"errorMessage":"Incorrect declared batch size, premature EOF reached","messagePattern":"Incorrect declared batch size, premature EOF reached","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java","lineNumber":308,"sourceCode":"            return new StreamRecordIterator(inputStream) {\n                @Override\n                protected Record doReadRecord(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) throws IOException {\n                    return DefaultRecord.readFrom(inputStream, baseOffset, baseTimestamp, baseSequence, logAppendTime);\n                }\n            };\n        }\n    }\n\n    private CloseableIterator<Record> uncompressedIterator() {\n        final ByteBuffer buffer = this.buffer.duplicate();\n        buffer.position(RECORDS_OFFSET);\n        return new RecordIterator() {\n            @Override\n            protected Record readNext(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) {\n                try {\n                    return DefaultRecord.readFrom(buffer, baseOffset, baseTimestamp, baseSequence, logAppendTime);\n                } catch (BufferUnderflowException e) {\n                    throw new InvalidRecordException(\"Incorrect declared batch size, premature EOF reached\");\n                }\n            }\n            @Override\n            protected boolean ensureNoneRemaining() {\n                return !buffer.hasRemaining();\n            }\n            @Override\n            public void close() {}\n        };\n    }\n\n    @Override\n    public Iterator<Record> iterator() {\n        if (count() == 0)\n            return Collections.emptyIterator();\n\n        if (!isCompressed())\n            return uncompressedIterator();","sourceCodeStart":290,"sourceCodeEnd":326,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java#L290-L326","documentation":"Thrown by the uncompressed batch iterator when readNext throws BufferUnderflowException — i.e. the batch header declared more records than the remaining bytes can hold, so decoding a record ran off the end of the ByteBuffer. InvalidRecordException, surfaced to consumer/fetch and log-iteration callers.","triggerScenarios":"Produced when iterating a DefaultRecordBatch whose declared record count / length overstates the bytes actually present: a torn write that updated length last, a wrong sizeInBytes passed to writeHeader, or a buffer truncated between writing the header and the records. Caught specifically at DefaultRecordBatch.java:307-309 around DefaultRecord.readFrom.","commonSituations":"Broker crash mid-flush leaving the segment's trailing batch with a header claiming N records but fewer bytes present; client fetching with a max.partition.fetch.bytes that sliced the batch mid-way (should normally not happen because batches are read atomically, but a mis-sized buffer in custom code can do it); a producer that computed sizeInBytes incorrectly (off-by-overhead) before writeHeader.","solutions":["Confirm with kafka-dump-log that the segment's last batch is truncated; if so the broker log recovery should truncate it — verify recovery completed and check the controller log for truncation messages.","If you build batches manually, ensure sizeInBytes passed to DefaultRecordBatch.writeHeader equals RECORD_BATCH_OVERHEAD + sum of DefaultRecord.sizeInBytes for each record (use the builder rather than hand-computing).","Raise max.partition.fetch.bytes (and fetch.max.bytes) if consumers are slicing batches; the broker never returns a partial batch to a compliant client, so this mainly affects custom readers.","Investigate unclean shutdowns: ensure log.flush.interval.messages / log.flush.interval.ms and replication factor keep in-sync replicas so an unclean leader election cannot expose a torn segment."],"exampleFix":"// before: hand-computing size and undercounting\nint size = records.stream().mapToInt(r -> r.sizeInBytes()).sum(); // missing overhead\nwriteHeader(buf, baseOffset, delta, size, magic, ...);\n\n// after: let the builder compute size including the batch overhead\ntry (MemoryRecordsBuilder b = MemoryRecords.builder(buf,\n        RecordBatch.CURRENT_MAGIC_VALUE, compression, TimestampType.CREATE_TIME, baseOffset)) {\n    records.forEach(b::append);\n}","handlingStrategy":"try-catch","validationCode":"// Thrown when iterating a batch whose declared length overshoots the underlying buffer.\n// Cannot be pre-validated at the API boundary — the declared size is internal to the batch.\n// Defensive habit: do not retain/reuse ByteBuffer slices across batches after partial reads.","typeGuard":null,"tryCatchPattern":"// Wrapped in the batch iterator; isolate and advance:\ntry {\n    batch.forEach(this::process);\n} catch (InvalidRecordException e) { // premature EOF\n    log.warn(\"Batch at {} @ {} declared more bytes than present\", partition, offset, e);\n    consumer.seek(partition, offset + 1);\n}","preventionTips":["Frequently caused by fetching with fetch.max.bytes too small for large batches — raise fetch.max.bytes and max.partition.fetch.bytes if you see this on big-payload topics.","Do not slice/compact the fetched ByteBuffer before the consumer is done with it; relative positions get out of sync and surface as premature EOF.","If mirroring with MirrorMaker, ensure message.max.bytes matches between source and target to avoid truncation mid-batch."],"tags":["kafka","record-format","consumer","broker","corruption","buffer"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}