apache/kafka · error · CorruptRecordException

Record batch for partition {} at offset {} is invalid, cause

Error message

Record batch for partition {} at offset {} is invalid, cause: {}

What it means

CorruptRecordException re-thrown from ShareCompletedFetch.maybeEnsureValid when a fetched RecordBatch fails batch.ensureValid() with checkCrcs enabled. The message decorates the underlying cause with the topic-partition and the batch baseOffset so operators can localize the corruption. It indicates the on-wire batch did not pass CRC/size validation, i.e. the bytes the consumer received are not the bytes the broker stored.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:420

                    lastRecord = record;
                    break;
                }
            }
        }

        return records != null && records.hasNext();
    }

    private Optional<Integer> maybeLeaderEpoch(final int leaderEpoch) {
        return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
    }

    private void maybeEnsureValid(RecordBatch batch, boolean checkCrcs) {
        if (checkCrcs && batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
            try {
                batch.ensureValid();
            } catch (CorruptRecordException e) {
                throw new CorruptRecordException("Record batch for partition " + partition.topicPartition()
                        + " at offset " + batch.baseOffset() + " is invalid, cause: " + e.getMessage());
            }
        }
    }

    private void maybeEnsureValid(final Record record, final boolean checkCrcs) {
        if (checkCrcs) {
            try {
                record.ensureValid();
            } catch (CorruptRecordException e) {
                throw new CorruptRecordException("Record for partition " + partition.topicPartition()
                        + " at offset " + record.offset() + " is invalid, cause: " + e.getMessage());
            }
        }
    }

    private void maybeCloseRecordStream() {
        if (records != null) {

View on GitHub (pinned to c31c9215e1)

Solutions

Retry the poll; transient corruption in transit often clears on the next fetch.
If persistent, set check.crcs=false only to bypass (not recommended long-term) and verify with a fresh fetch from a different broker/replica.
Inspect broker logs and disk health for the affected partition/offset range; restore the segment from a healthy replica or recreate the topic data.
Confirm broker and client versions match to rule out a record-format incompatibility.

Defensive patterns

Strategy: try-catch

Validate before calling

// No pre-call validation possible: corruption is detected by CRC at read time.
// The only knob is whether the client checks at all.
props.put("check.crcs", "false"); // disables this check entirely (trade-off documented below)

Try / catch

// CorruptRecordException is recoverable: skip and acknowledge, do not block the share group.
import org.apache.kafka.common.errors.CorruptRecordException;

try {
    ConsumerRecords<K,V> records = shareConsumer.poll(Duration.ofSeconds(5));
    // ... process ...
} catch (CorruptRecordException e) {
    // The fetch's partition/offset is in the message; acknowledge to advance.
    log.error("Corrupt batch detected, skipping: {}", e.getMessage(), e);
    shareConsumer.acknowledge(e.offsetAndMetadata()); // or context-appropriate ack
}

Prevention

Corruption usually means disk/network/broker faults — surface it to ops, do not swallow it silently.
Set check.crcs=true in production (the default); only disable in trusted LANs where CPU matters more than silent corruption.
Track corrupt-offset metrics so you can correlate with broker disk failures or producer bugs.
Wire CorruptRecordException to your dead-letter / skip pipeline so the share group keeps progressing.

When it happens

Trigger: Polling a share consumer that fetches a batch whose magic is >= V2 while 'check.crcs' is true (default) and batch.ensureValid() throws CorruptRecordException. Reached on every iteration inside the fetch streaming loop of ShareCompletedFetch.

Common situations: Disk or page-cache bit rot on the broker, network/SSL corruption in transit, a transient OS-level I/O error on the broker log segment, or a client pointed at a stale/failing broker replica. Can also appear after an unclean leader election or partial segment truncation where headers and payload disagree.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/9dab223b96642793.json. Report an issue: GitHub.