apache/kafka · error · InvalidRecordException

Incorrect declared batch size, records still remaining in fi

Error message

Incorrect declared batch size, records still remaining in file

What it means

Thrown by DefaultRecordBatch.RecordIterator.next() when, after iterating the declared record count for the batch, ensureNoneRemaining() reports bytes still left in the underlying buffer/stream. It is an integrity check that the batch's declared record count matches the actual payload length, guarding against truncated or padded batches whose size header is inconsistent with the serialized records. The library throws InvalidRecordException (not a generic IOException) so callers can classify it as a corrupt-record condition and skip/quarantine the batch rather than retry blindly.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java:608

        @Override
        public boolean hasNext() {
            return readRecords < numRecords;
        }

        @Override
        public Record next() {
            if (readRecords >= numRecords)
                throw new NoSuchElementException();

            readRecords++;
            Record rec = readNext(baseOffset, baseTimestamp, baseSequence, logAppendTime);
            if (readRecords == numRecords) {
                // Validate that the actual size of the batch is equal to declared size
                // by checking that after reading declared number of items, there no items left
                // (overflow case, i.e. reading past buffer end is checked elsewhere).
                if (!ensureNoneRemaining())
                    throw new InvalidRecordException("Incorrect declared batch size, records still remaining in file");
            }
            return rec;
        }

        protected abstract Record readNext(long baseOffset, long baseTimestamp, int baseSequence, Long logAppendTime);

        protected abstract boolean ensureNoneRemaining();

        @Override
        public void remove() {
            throw new UnsupportedOperationException();
        }

    }

    // visible for testing
    abstract class StreamRecordIterator extends RecordIterator {
        private final InputStream inputStream;

View on GitHub (pinned to c31c9215e1)

Solutions

Inspect the segment with kafka-dump-log.sh (--index-sanity-check true --deep-iteration) on the affected .log file to confirm the batch with the mismatched count and its offset.
If corruption is isolated, truncate the segment past the last valid offset and let replication/leadership migration re-replicate from an in-sync replica; verify the topic's min.insync.replicas before doing so.
If the count mismatch is from a producer/serializer bug, reproduce locally with a unit test asserting batch.count() against the serialized payload, fix the writer, and re-produce the affected messages.
Enable log.flush.interval.messages / log.flush.interval.ms tuning only if the root cause was a torn write from improper flush/fsync behavior; otherwise leave defaults.

Example fix

// before: custom batch writer that writes records then patches a wrong count
int declared = buffer.getInt(COUNT_OFFSET);
// after: compute count from actual records written, write once
buffer.putInt(COUNT_OFFSET, recordsWritten);

Defensive patterns

Strategy: try-catch

Try / catch

try {
  for (Record r : batch) { /* process */ }
} catch (org.apache.kafka.common.InvalidRecordException e) {
  // Declared record count under-counts actual bytes: batch is corrupt.
  // Quarantine the segment, record batch.baseOffset(), skip to next batch.
  log.warn("Corrupt batch at offset {}: size mismatch", batch.baseOffset(), e);
}

Prevention

Treat InvalidRecordException as unrecoverable for that batch: do not retry the same bytes, skip or seek past the batch.
Enable broker-side CRC32C validation (unclean.leader.shutdown / log validation) so corrupt batches are rejected before persisting.
Run kafka-dump-log --files <segment> --deep-iteration to audit segments suspected of corruption before consumer reads.
Ensure producers wait for full acks (acks=all) and the broker fsyncs to avoid partial/truncated batch writes on crash.
Monitor disk health (SMART errors, EIO) — size-mismatch corruption is most often a storage or OS-crash artifact.

When it happens

Trigger: Iterating a DefaultRecordBatch (v2 magic) via its RecordIterator / streamingIterator where the batch's count() field undercounts the records actually encoded in the payload. Produced by calling forEach/iterator on a Records batch read from FileRecords or MemoryRecords, then the last next() triggers ensureNoneRemaining() which returns false because inputStream.read() != -1.

Common situations: On-disk log segment corruption (partial write, torn write after a broker crash, fsync gap), a bug in a custom serializer or producer interceptor that mis-writes the record count, reading a log file produced by an incompatible/forked Kafka build, or a partial segment recovery where the batch footer/size was rewritten but the count was not. Also seen after unsafe manual edits to .log segment files or when two segments were concatenated incorrectly.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/0e3d25b9e8cbc98e.json. Report an issue: GitHub.