apache/kafka · error · CorruptRecordException

Record size %d exceeds the largest allowable message size (%

Error message

Record size %d exceeds the largest allowable message size (%d).

What it means

Thrown by ByteBufferLogInputStream.nextBatchSize when the record size decoded from the header exceeds the maxMessageSize limit the stream was constructed with. The cap is the broker/client configured message ceiling (message.max.bytes / max.message.bytes); a record claiming to be larger is either corrupt or indicates the producer and reader disagree on the configured limit. Treated as CorruptRecordException because a valid stream cannot contain a batch larger than the negotiated maximum.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/ByteBufferLogInputStream.java:76

    }

    /**
     * Validates the header of the next batch and returns batch size.
     * @return next batch size including LOG_OVERHEAD if buffer contains header up to
     *         magic byte, null otherwise
     * @throws CorruptRecordException if record size or magic is invalid
     */
    Integer nextBatchSize() throws CorruptRecordException {
        int remaining = buffer.remaining();
        if (remaining < LOG_OVERHEAD)
            return null;
        int recordSize = buffer.getInt(buffer.position() + SIZE_OFFSET);
        // V0 has the smallest overhead, stricter checking is done later
        if (recordSize < LegacyRecord.RECORD_OVERHEAD_V0)
            throw new CorruptRecordException(String.format("Record size %d is less than the minimum record overhead (%d)",
                    recordSize, LegacyRecord.RECORD_OVERHEAD_V0));
        if (recordSize > maxMessageSize)
            throw new CorruptRecordException(String.format("Record size %d exceeds the largest allowable message size (%d).",
                    recordSize, maxMessageSize));

        if (remaining < HEADER_SIZE_UP_TO_MAGIC)
            return null;

        byte magic = buffer.get(buffer.position() + MAGIC_OFFSET);
        if (magic < 0 || magic > RecordBatch.CURRENT_MAGIC_VALUE)
            throw new CorruptRecordException("Invalid magic found in record: " + magic);

        return recordSize + LOG_OVERHEAD;
    }
}

View on GitHub (pinned to c31c9215e1)

Solutions

Align max message size across producer, broker (message.max.bytes), topic (max.message.bytes), and consumer (max.partition.fetch.bytes / fetch.max.bytes), then restart the affected component.
Verify with kafka-dump-log whether the size field reflects a real large batch (config mismatch) or an implausible value (corruption).
If corrupt, delete/restore the offending segment as in the underflow case.

Example fix

# before
message.max.bytes=1000012

# after (match across producer/broker/topic/consumer)
message.max.bytes=8388608
max.message.bytes=8388608
max.request.size=8388608
fetch.max.bytes=8388608

Defensive patterns

Strategy: try-catch

Validate before calling

// The limit comes from maxMessageSize passed to ByteBufferLogInputStream.
// Ensure it matches broker/message max (message.max.bytes / max.message.bytes):
int maxMessageSize = Math.max(configuredMax, recordSize + LOG_OVERHEAD);
// If you already know recordSize, you can short-circuit:
if (recordSize > maxMessageSize) {
    // oversized or corrupt; do not attempt to materialize
}

Try / catch

try {
    batch = stream.nextBatch();
} catch (org.apache.kafka.common.errors.CorruptRecordException e) {
    // either a genuinely oversized message or corruption;
    // check message.max.bytes on producer, broker, and consumer configs
}

Prevention

Align max.message.bytes across producer, broker, and consumer to avoid truncation reads.
A size above maxMessageSize usually means the producer was misconfigured or the segment is corrupt; investigate the source.
Cap serialized record size client-side before send() to fail fast instead of on read.

When it happens

Trigger: nextBatchSize reads a size value greater than the maxMessageSize passed into ByteBufferLogInputStream's constructor. Happens during fetch/replication/log-recovery when the producing side wrote a batch larger than the reading side's maxMessageSize, or when the size field is itself corrupt.

Common situations: Broker A has message.max.bytes=2MB and produces a large batch; broker B / consumer / replica is configured with a smaller max and rejects it. Also seen when a size field is bit-flipped/corrupt, inflating the value.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/caddee42af2d1fde.json. Report an issue: GitHub.