apache/kafka · critical · CorruptRecordException

Record size %d is less than the minimum record overhead (%d)

Error message

Record size %d is less than the minimum record overhead (%d)

What it means

Thrown by ByteBufferLogInputStream.nextBatchSize when the size read from the batch header (buffer position + SIZE_OFFSET) is smaller than LegacyRecord.RECORD_OVERHEAD_V0 — the absolute minimum byte overhead a v0 record can occupy. A size below this floor is structurally impossible for any valid record, so the stream treats the segment as corrupt and raises CorruptRecordException. V0 has the smallest overhead, so anything smaller is unambiguously garbage.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/ByteBufferLogInputStream.java:73

            return new DefaultRecordBatch(batchSlice);
        else
            return new AbstractLegacyRecordBatch.ByteBufferLegacyRecordBatch(batchSlice);
    }

    /**
     * Validates the header of the next batch and returns batch size.
     * @return next batch size including LOG_OVERHEAD if buffer contains header up to
     *         magic byte, null otherwise
     * @throws CorruptRecordException if record size or magic is invalid
     */
    Integer nextBatchSize() throws CorruptRecordException {
        int remaining = buffer.remaining();
        if (remaining < LOG_OVERHEAD)
            return null;
        int recordSize = buffer.getInt(buffer.position() + SIZE_OFFSET);
        // V0 has the smallest overhead, stricter checking is done later
        if (recordSize < LegacyRecord.RECORD_OVERHEAD_V0)
            throw new CorruptRecordException(String.format("Record size %d is less than the minimum record overhead (%d)",
                    recordSize, LegacyRecord.RECORD_OVERHEAD_V0));
        if (recordSize > maxMessageSize)
            throw new CorruptRecordException(String.format("Record size %d exceeds the largest allowable message size (%d).",
                    recordSize, maxMessageSize));

        if (remaining < HEADER_SIZE_UP_TO_MAGIC)
            return null;

        byte magic = buffer.get(buffer.position() + MAGIC_OFFSET);
        if (magic < 0 || magic > RecordBatch.CURRENT_MAGIC_VALUE)
            throw new CorruptRecordException("Invalid magic found in record: " + magic);

        return recordSize + LOG_OVERHEAD;
    }
}

View on GitHub (pinned to c31c9215e1)

Solutions

Delete the corrupt segment (.log + .index/.timeindex) and let Kafka recover, or restore from a known-good replica.
Run kafka-dump-log --files <segment.log> --print-data-log to inspect the offending batch and confirm the size field.
Ensure the broker shut down cleanly and the disk/fs is healthy; check dmesg / filesystem logs for underlying I/O errors.

Defensive patterns

Strategy: try-catch

Validate before calling

// This is raised on read from a (corrupt) byte stream; you cannot pre-validate
// without parsing. If you control the buffer, sanity-check the size field:
int recordSize = buffer.getInt(buffer.position() + SIZE_OFFSET);
if (recordSize < LegacyRecord.RECORD_OVERHEAD_V0) {
    // corrupt/truncated log segment; stop iterating
}

Try / catch

try {
    batch = stream.nextBatch();
} catch (org.apache.kafka.common.errors.CorruptRecordException e) {
    // log segment is corrupt; skip batch, advance position, or halt recovery
    log.warn("Corrupt record (size below overhead), skipping", e);
}

Prevention

Treat CorruptRecordException as a symptom of disk/replication corruption, not caller error.
Validate checksums / use replication to detect bit-rot before reading.
When iterating a log, be prepared to skip past corrupt batches rather than aborting the whole scan.

When it happens

Trigger: Reading a log segment via ByteBufferLogInputStream where the 4-byte size field decodes to a value below LegacyRecord.RECORD_OVERHEAD_V0. Produced by FileLogInputStream/byte-buffer scanning during fetch, replication, or log recovery.

Common situations: On-disk log corruption (partial write, fsync failure, disk fault, unclean shutdown); truncated segment tail; reading a file that is not actually a Kafka log; mismatched segment offset/index files after a manual copy.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/c3169dd35c62530c.json. Report an issue: GitHub.