apache/kafka · error · InvalidRecordException

Found invalid record count {} in magic v{} batch

Error message

Found invalid record count {} in magic v{} batch

What it means

Thrown by DefaultRecordBatch.RecordIterator's constructor when the batch's declared record count (read via count() at the COUNT_OFFSET field) is negative. A negative count is structurally invalid — the count field encodes a non-negative varint/int of records — so this indicates either a corrupt batch or a buffer that was not actually a v2 batch. InvalidRecordException during iteration.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java:586

    }

    // visible for testing
    abstract class RecordIterator implements CloseableIterator<Record> {
        private final Long logAppendTime;
        private final long baseOffset;
        private final long baseTimestamp;
        private final int baseSequence;
        private final int numRecords;
        private int readRecords = 0;

        RecordIterator() {
            this.logAppendTime = timestampType() == TimestampType.LOG_APPEND_TIME ? maxTimestamp() : null;
            this.baseOffset = baseOffset();
            this.baseTimestamp = baseTimestamp();
            this.baseSequence = baseSequence();
            int numRecords = count();
            if (numRecords < 0)
                throw new InvalidRecordException("Found invalid record count " + numRecords + " in magic v" +
                        magic() + " batch");
            this.numRecords = numRecords;
        }

        @Override
        public boolean hasNext() {
            return readRecords < numRecords;
        }

        @Override
        public Record next() {
            if (readRecords >= numRecords)
                throw new NoSuchElementException();

            readRecords++;
            Record rec = readNext(baseOffset, baseTimestamp, baseSequence, logAppendTime);
            if (readRecords == numRecords) {
                // Validate that the actual size of the batch is equal to declared size

View on GitHub (pinned to c31c9215e1)

Solutions

1. Call DefaultRecordBatch.ensureValid() before iterating; it catches size/CRC corruption earlier and isolates the bad batch.
2. Verify the buffer is actually a v2 batch (magic == 2 at MAGIC_OFFSET) before constructing DefaultRecordBatch.
3. If the segment is corrupt at the tail, recover via the broker's log-recovery truncation or remove the affected segment file.
4. In custom readers, do not advance past a batch whose length/count fields are inconsistent — re-fetch or skip rather than iterate.

Example fix

// before: iterating an unvalidated, possibly-non-batch buffer
DefaultRecordBatch b = new DefaultRecordBatch(buf);
for (Record r : b) { ... }

// after: validate before iteration
DefaultRecordBatch b = new DefaultRecordBatch(buf);
if (b.magic() != RecordBatch.CURRENT_MAGIC_VALUE) throw new IOException("not a v2 batch");
b.ensureValid();
for (Record r : b) { ... }

// Record count is read out of the batch header on the READ path; user code does not pass it.
// The only prevention on the produce side is to never forge batches with negative counts —
// always use MemoryRecordsBuilder, which computes count correctly.

// Invalid record count surfaces when iterating; skip and continue:
try {
    batch.iterator().forEachRemaining(this::process);
} catch (InvalidRecordException e) { // negative record count
    log.error("Batch at {} @ {} has invalid record count, skipping", partition, offset, e);
    consumer.seek(partition, offset + 1);
}

When it happens

Trigger: Produced when iterator()/streamingIterator() is called on a DefaultRecordBatch whose COUNT_OFFSET int decodes to a negative value — e.g. reading a buffer whose header was zeroed/garbage, a partial write that left the count field unset, or pointing a DefaultRecordBatch at a ByteBuffer that is not a batch at all. The check at DefaultRecordBatch.java:585-587 precedes any record decoding.

Common situations: Log recovery scanning a segment with a torn or zeroed trailing header; consumer fetching from a replica that served a partially-written batch; custom tooling that wraps arbitrary ByteBuffers in DefaultRecordBatch without prior ensureValid; downgrade/upgrade where the count field semantics were misread.

Related errors

• Incorrect declared batch size, premature EOF reached
• Record batch is corrupt (the size {} is smaller than the min
• Record is corrupt (stored crc = {}, computed crc = {})
• Timestamp type must be provided to compute attributes for me
• The target time for partition {} is {}. The target time cann

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/9d2c5632c310e1b2.json. Report an issue: GitHub.