apache/kafka · critical · CorruptRecordException

Record batch is corrupt (the size {} is smaller than the min

Error message

Record batch is corrupt (the size {} is smaller than the minimum allowed overhead {})

What it means

Thrown by DefaultRecordBatch.ensureValid when the batch's total sizeInBytes is smaller than RECORD_BATCH_OVERHEAD (the fixed header bytes before any records: base offset, length, leader epoch, magic, CRC, producer id/epoch, sequence, timestamps, etc.). A batch that short cannot contain a valid header, so it is treated as structural corruption. CorruptRecordException.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java:153

    private static final int CONTROL_FLAG_MASK = 0x20;
    private static final byte DELETE_HORIZON_FLAG_MASK = 0x40;
    private static final byte TIMESTAMP_TYPE_MASK = 0x08;

    private final ByteBuffer buffer;

    DefaultRecordBatch(ByteBuffer buffer) {
        this.buffer = buffer;
    }

    @Override
    public byte magic() {
        return buffer.get(MAGIC_OFFSET);
    }

    @Override
    public void ensureValid() {
        if (sizeInBytes() < RECORD_BATCH_OVERHEAD)
            throw new CorruptRecordException("Record batch is corrupt (the size " + sizeInBytes() +
                    " is smaller than the minimum allowed overhead " + RECORD_BATCH_OVERHEAD + ")");

        if (!isValid())
            throw new CorruptRecordException("Record is corrupt (stored crc = " + checksum()
                    + ", computed crc = " + computeChecksum() + ")");
    }

    /**
     * Gets the base timestamp of the batch which is used to calculate the record timestamps from the deltas.
     *
     * @return The base timestamp
     */
    public long baseTimestamp() {
        return buffer.getLong(BASE_TIMESTAMP_OFFSET);
    }

    @Override
    public long maxTimestamp() {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Run kafka-dump-log --files <segment-log> to confirm which batch/offset is short and whether the segment tail is truncated.
2. If the segment is torn at the tail, delete the truncated segment (or use kafka-storage/kafka-clean tooling) so recovery skips the remnant; the broker will truncate to the last complete batch.
3. Ensure client/broker code reads whole batches via MemoryRecords rather than slicing at computed offsets; never allocate a batch buffer smaller than RECORD_BATCH_OVERHEAD.
4. Verify broker log.flush / log.segment configuration is not producing partial flushes; check disk health (df, dmesg) for I/O errors.

Example fix

// before: slicing a buffer to a length smaller than the batch header
DefaultRecordBatch b = new DefaultRecordBatch(buf.slice(0, partialLen));
b.ensureValid();

// after: only treat complete batches as batches
if (readable >= DefaultRecordBatch.RECORD_BATCH_OVERHEAD) {
    DefaultRecordBatch b = new DefaultRecordBatch(buf);
    b.ensureValid();
}

// Thrown from DefaultRecordBatch.ensureValid() while READING a buffer; the user does not
// supply 'size' as an argument. The only pre-call defense is to reject undersized buffers
// before handing them to the batch reader:
void checkBatch(ByteBuffer b) {
    if (b == null || b.remaining() < DefaultRecordBatch.RECORD_BATCH_OVERHEAD) {
        throw new CorruptRecordException("buffer too small to be a batch: " + (b == null ? -1 : b.remaining()));
    }
}

// Narrow 'raw bytes' to a validated batch wrapper before use:
static final class VerifiedBatch {
    private final DefaultRecordBatch batch;
    private VerifiedBatch(DefaultRecordBatch b) { this.batch = b; }
    static VerifiedBatch of(ByteBuffer buf) {
        if (buf.remaining() < DefaultRecordBatch.RECORD_BATCH_OVERHEAD)
            throw new CorruptRecordException("undersized");
        return new VerifiedBatch(new DefaultRecordBatch(buf));
    }
    DefaultRecordBatch get() { return batch; }
}

// On consume, isolate the corrupt batch and continue:
try {
    batch.ensureValid();
    process(batch);
} catch (CorruptRecordException e) { // size < RECORD_BATCH_OVERHEAD
    log.warn("Truncated/corrupt batch at {} offset {}, skipping", partition, offset, e);
    consumer.seek(partition, offset + 1);
}

When it happens

Trigger: Produced when a buffer sliced shorter than the batch overhead is presented to a DefaultRecordBatch and ensureValid() is called (broker append path, consumer validation, log recovery). Often follows an undersized ByteBuffer allocation, a partial read off disk/network, or a torn write leaving a sub-overhead remnant at a segment tail.

Common situations: Truncated log segment after an unclean broker shutdown; undersized buffer in custom code that reads exactly fetch.minBytes / a miscomputed length; inter-broker or client fetch of a segment whose last batch was only partially flushed; OS page-cache vs disk inconsistency after a crash.

Related errors

• Record is corrupt (stored crc = {}, computed crc = {})
• Incorrect declared batch size, premature EOF reached
• Found invalid record count {} in magic v{} batch
• Timestamp type must be provided to compute attributes for me
• Incorrect declared batch size, records still remaining in fi

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/427b0c06804db8ee.json. Report an issue: GitHub.