apache/kafka · critical · CorruptRecordException

Record size is less than the minimum record overhead (%d)

Error message

Record size is less than the minimum record overhead (%d)

What it means

Thrown by DataLogInputStream.nextBatch() (line 301) as a CorruptRecordException when the size field read from a legacy (magic 0/1) message frame is smaller than LegacyRecord.RECORD_OVERHEAD_V0. The declared size is too small to even hold the fixed legacy record header, so the byte stream cannot contain a well-formed record. Kafka treats this as a corrupt-log condition that should cause the consumer to skip or fail the fetch depending on the broker's handling.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java:301

        private final int maxMessageSize;
        private final ByteBuffer offsetAndSizeBuffer;

        DataLogInputStream(InputStream stream, int maxMessageSize) {
            this.stream = stream;
            this.maxMessageSize = maxMessageSize;
            this.offsetAndSizeBuffer = ByteBuffer.allocate(Records.LOG_OVERHEAD);
        }

        public AbstractLegacyRecordBatch nextBatch() throws IOException {
            offsetAndSizeBuffer.clear();
            Utils.readFully(stream, offsetAndSizeBuffer);
            if (offsetAndSizeBuffer.hasRemaining())
                return null;

            long offset = offsetAndSizeBuffer.getLong(Records.OFFSET_OFFSET);
            int size = offsetAndSizeBuffer.getInt(Records.SIZE_OFFSET);
            if (size < LegacyRecord.RECORD_OVERHEAD_V0)
                throw new CorruptRecordException(String.format("Record size is less than the minimum record overhead (%d)", LegacyRecord.RECORD_OVERHEAD_V0));
            if (size > maxMessageSize)
                throw new CorruptRecordException(String.format("Record size exceeds the largest allowable message size (%d).", maxMessageSize));

            ByteBuffer batchBuffer = ByteBuffer.allocate(size);
            Utils.readFully(stream, batchBuffer);
            if (batchBuffer.hasRemaining())
                return null;
            batchBuffer.flip();

            return new BasicLegacyRecordBatch(offset, new LegacyRecord(batchBuffer));
        }
    }

    private static class DeepRecordsIterator extends AbstractIterator<Record> implements CloseableIterator<Record> {
        private final ArrayDeque<AbstractLegacyRecordBatch> innerEntries;
        private final long absoluteBaseOffset;
        private final byte wrapperMagic;

View on GitHub (pinned to c31c9215e1)

Solutions

1. Run kafka-dump-log on the suspect segment to confirm corruption and locate the torn record.
2. Delete or truncate the corrupt segment (stop the broker, move the .log/.index/.timeindex aside, restart) so replication can rebuild it.
3. If reading from a ByteBuffer you constructed, verify the framing logic against Records.LOG_OVERHEAD + LegacyRecord.RECORD_OVERHEAD_V0 before calling nextBatch().
4. Ensure brokers shut down cleanly and use replication factor >= 2 so a healthy replica can replace the corrupt one.

// CorruptRecordException is thrown while iterating a legacy (v0/v1) record stream.
// You cannot pre-validate byte sizes without reading the stream, so wrap iteration.
import org.apache.kafka.common.errors.CorruptRecordException;

try {
    for (RecordBatch batch : records.batches()) {
        for (Record r : batch) { /* process */ }
    }
} catch (CorruptRecordException e) {
    // record is shorter than RECORD_OVERHEAD_V0 => truncated/garbled segment.
    // quarantine the segment, advance the consumer past it, or alert.
    log.error("Corrupt legacy record (size < min overhead) in {}", topicPartition, e);
}

When it happens

Trigger: Reading a corrupted or partially-written log segment through FileRecords / a LogInputStream, or feeding a truncated ByteBuffer into a legacy record iterator. Common when a broker crashed mid-flush, when on-disk data was truncated externally, or when an off-by-one frame parser feeds DataLogInputStream garbage.

Common situations: Broker crash or power loss during log append leaving a torn last record. Disk corruption or filesystem-level truncation of a .log segment. Custom producers that hand-craft message frames with a wrong size field. Mixing non-Kafka bytes into a file treated as a Kafka log.

Related errors

• Record batch for partition {} at offset {} is invalid, cause
• Record for partition {} at offset {} is invalid, cause: {}
• Encountered corrupt message when fetching offset {} for topi
• Record size exceeds the largest allowable message size (%d).
• Incorrect declared batch size, premature EOF reached

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/f586961c70754365.json. Report an issue: GitHub.