apache/kafka · error · KafkaException

Record batch for partition {} at offset {} is invalid, cause

Error message

Record batch for partition {} at offset {} is invalid, cause: {}

What it means

KafkaException wrapping a CorruptRecordException, thrown by CompletedFetch.maybeEnsureValid(FetchConfig, RecordBatch) at CompletedFetch.java:163 when check.crcs is enabled (default true) and a v2+ record batch fails its CRC32C check via batch.ensureValid(). The CRC covers the entire batch payload, so a mismatch means the batch was altered after the producer wrote it — typically disk or network corruption. The original CorruptRecordException's message is appended as the cause.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/internals/CompletedFetch.java:163

        if (!isConsumed) {
            maybeCloseRecordStream();
            cachedRecordException = null;
            this.isConsumed = true;
            recordAggregatedMetrics(bytesRead, recordsRead);

            // we move the partition to the end if we received some bytes. This way, it's more likely that partitions
            // for the same topic can remain together (allowing for more efficient serialization).
            if (bytesRead > 0)
                subscriptions.movePartitionToEnd(partition);
        }
    }

    private void maybeEnsureValid(FetchConfig fetchConfig, RecordBatch batch) {
        if (fetchConfig.checkCrcs && batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
            try {
                batch.ensureValid();
            } catch (CorruptRecordException e) {
                throw new KafkaException("Record batch for partition " + partition + " at offset " +
                        batch.baseOffset() + " is invalid, cause: " + e.getMessage());
            }
        }
    }

    private void maybeEnsureValid(FetchConfig fetchConfig, Record record) {
        if (fetchConfig.checkCrcs) {
            try {
                record.ensureValid();
            } catch (CorruptRecordException e) {
                throw new KafkaException("Record for partition " + partition + " at offset " + record.offset()
                        + " is invalid, cause: " + e.getMessage());
            }
        }
    }

    private void maybeCloseRecordStream() {
        if (records != null) {

View on GitHub (pinned to c31c9215e1)

Solutions

Verify broker disk health and inspect broker logs for write/checksum errors; a hardware fault is the most common root cause.
Skip the corrupt batch by seeking past it: consumer.seek(partition, batchEndOffset + 1), accepting loss of that batch's records.
If possible, force leadership to a healthy replica and let the consumer re-fetch from there.
Use kafka-dump-log on the broker segment to confirm corruption and decide whether to restore from snapshot or delete the bad segment.

Defensive patterns

Strategy: try-catch

Try / catch

try {
    ConsumerRecords<K, V> recs = consumer.poll(Duration.ofSeconds(1));
} catch (KafkaException e) {
    if (e.getCause() instanceof CorruptRecordException || e.getMessage().contains("is invalid")) {
        TopicPartition tp = affectedPartition;          // track from records/partition
        long next = consumer.position(tp);              // baseOffset of bad batch
        consumer.seek(tp, next + 1);                     // skip past corrupt batch
    } else throw e;
}

Prevention

Keep check.crcs = true (the default) so corruption is detected rather than silently consumed.
On a corrupt-batch error, seek past baseOffset+1 to resume; do not retry the same offset.
Recurring CRC failures indicate disk or network faults — investigate the broker/hardware.
Never disable CRC checking in production to 'make the error go away'.

When it happens

Trigger: consumer.poll(...) with check.crcs=true (the default) returns a record batch whose recomputed CRC32C does not match the stored value. Validation runs from nextFetchedRecord at CompletedFetch.java:206 each time a new batch is taken from the iterator.

Common situations: Broker disk or page-cache bit-rot; faulty NIC, cable, switch, or RAM flipping bits in transit; partial segment writes after a broker crash; rare client/broker message-format skew; hardware fault on the producing path that wrote a bad batch.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/ee1f76b3cc7669d2.json. Report an issue: GitHub.