apache/kafka · error · InvalidRecordException

Incorrect declared batch size, premature EOF reached

Error message

Incorrect declared batch size, premature EOF reached

What it means

Thrown by the uncompressed batch iterator when readNext throws BufferUnderflowException — i.e. the batch header declared more records than the remaining bytes can hold, so decoding a record ran off the end of the ByteBuffer. InvalidRecordException, surfaced to consumer/fetch and log-iteration callers.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java:308

            return new StreamRecordIterator(inputStream) {
                @Override
                protected Record doReadRecord(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) throws IOException {
                    return DefaultRecord.readFrom(inputStream, baseOffset, baseTimestamp, baseSequence, logAppendTime);
                }
            };
        }
    }

    private CloseableIterator<Record> uncompressedIterator() {
        final ByteBuffer buffer = this.buffer.duplicate();
        buffer.position(RECORDS_OFFSET);
        return new RecordIterator() {
            @Override
            protected Record readNext(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime) {
                try {
                    return DefaultRecord.readFrom(buffer, baseOffset, baseTimestamp, baseSequence, logAppendTime);
                } catch (BufferUnderflowException e) {
                    throw new InvalidRecordException("Incorrect declared batch size, premature EOF reached");
                }
            }
            @Override
            protected boolean ensureNoneRemaining() {
                return !buffer.hasRemaining();
            }
            @Override
            public void close() {}
        };
    }

    @Override
    public Iterator<Record> iterator() {
        if (count() == 0)
            return Collections.emptyIterator();

        if (!isCompressed())
            return uncompressedIterator();

View on GitHub (pinned to c31c9215e1)

Solutions

Confirm with kafka-dump-log that the segment's last batch is truncated; if so the broker log recovery should truncate it — verify recovery completed and check the controller log for truncation messages.
If you build batches manually, ensure sizeInBytes passed to DefaultRecordBatch.writeHeader equals RECORD_BATCH_OVERHEAD + sum of DefaultRecord.sizeInBytes for each record (use the builder rather than hand-computing).
Raise max.partition.fetch.bytes (and fetch.max.bytes) if consumers are slicing batches; the broker never returns a partial batch to a compliant client, so this mainly affects custom readers.
Investigate unclean shutdowns: ensure log.flush.interval.messages / log.flush.interval.ms and replication factor keep in-sync replicas so an unclean leader election cannot expose a torn segment.

Example fix

// before: hand-computing size and undercounting
int size = records.stream().mapToInt(r -> r.sizeInBytes()).sum(); // missing overhead
writeHeader(buf, baseOffset, delta, size, magic, ...);

// after: let the builder compute size including the batch overhead
try (MemoryRecordsBuilder b = MemoryRecords.builder(buf,
        RecordBatch.CURRENT_MAGIC_VALUE, compression, TimestampType.CREATE_TIME, baseOffset)) {
    records.forEach(b::append);
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Thrown when iterating a batch whose declared length overshoots the underlying buffer.
// Cannot be pre-validated at the API boundary — the declared size is internal to the batch.
// Defensive habit: do not retain/reuse ByteBuffer slices across batches after partial reads.

Try / catch

// Wrapped in the batch iterator; isolate and advance:
try {
    batch.forEach(this::process);
} catch (InvalidRecordException e) { // premature EOF
    log.warn("Batch at {} @ {} declared more bytes than present", partition, offset, e);
    consumer.seek(partition, offset + 1);
}

Prevention

Frequently caused by fetching with fetch.max.bytes too small for large batches — raise fetch.max.bytes and max.partition.fetch.bytes if you see this on big-payload topics.
Do not slice/compact the fetched ByteBuffer before the consumer is done with it; relative positions get out of sync and surface as premature EOF.
If mirroring with MirrorMaker, ensure message.max.bytes matches between source and target to avoid truncation mid-batch.

When it happens

Trigger: Produced when iterating a DefaultRecordBatch whose declared record count / length overstates the bytes actually present: a torn write that updated length last, a wrong sizeInBytes passed to writeHeader, or a buffer truncated between writing the header and the records. Caught specifically at DefaultRecordBatch.java:307-309 around DefaultRecord.readFrom.

Common situations: Broker crash mid-flush leaving the segment's trailing batch with a header claiming N records but fewer bytes present; client fetching with a max.partition.fetch.bytes that sliced the batch mid-way (should normally not happen because batches are read atomically, but a mis-sized buffer in custom code can do it); a producer that computed sizeInBytes incorrectly (off-by-overhead) before writeHeader.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/a9d3ea55bceaea21.json. Report an issue: GitHub.