apache/kafka · error · KafkaException

Record for partition {} at offset {} is invalid, cause: {}

Error message

Record for partition {} at offset {} is invalid, cause: {}

What it means

KafkaException wrapping a CorruptRecordException, thrown by CompletedFetch.maybeEnsureValid(FetchConfig, Record) at CompletedFetch.java:174 when check.crcs is enabled and a single legacy-format (message magic v0/v1) record fails its per-record CRC check via record.ensureValid(). Unlike v2 batches where the CRC lives on the batch, pre-v2 messages carry a per-record checksum, so this path only fires on topics still stored in the old format.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/internals/CompletedFetch.java:174

    }

    private void maybeEnsureValid(FetchConfig fetchConfig, RecordBatch batch) {
        if (fetchConfig.checkCrcs && batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
            try {
                batch.ensureValid();
            } catch (CorruptRecordException e) {
                throw new KafkaException("Record batch for partition " + partition + " at offset " +
                        batch.baseOffset() + " is invalid, cause: " + e.getMessage());
            }
        }
    }

    private void maybeEnsureValid(FetchConfig fetchConfig, Record record) {
        if (fetchConfig.checkCrcs) {
            try {
                record.ensureValid();
            } catch (CorruptRecordException e) {
                throw new KafkaException("Record for partition " + partition + " at offset " + record.offset()
                        + " is invalid, cause: " + e.getMessage());
            }
        }
    }

    private void maybeCloseRecordStream() {
        if (records != null) {
            records.close();
            records = null;
        }
    }

    private Record nextFetchedRecord(FetchConfig fetchConfig) {
        while (true) {
            if (records == null || !records.hasNext()) {
                maybeCloseRecordStream();

                if (!batches.hasNext()) {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Skip the corrupt record: consumer.seek(partition, record.offset() + 1) and resume polling.
2. Inspect broker disk and network health; checksum failures almost always indicate hardware faults.
3. Plan a message-format upgrade to v2 once all clients are compatible, so per-record CRCs are replaced by batch-level coverage.
4. If the segment is unrecoverable, restore from backup or use kafka-dump-log to confirm before deleting the bad segment.

try {
    ConsumerRecords<K, V> recs = consumer.poll(Duration.ofSeconds(1));
} catch (KafkaException e) {
    if (e.getMessage().contains("Record for partition")) {
        TopicPartition tp = affectedPartition;
        long badOffset = consumer.position(tp);
        consumer.seek(tp, badOffset + 1);   // skip the single poison-pill record
    } else throw e;
}

When it happens

Trigger: consumer.poll(...) with check.crcs=true reading a legacy v0/v1 message whose individual CRC does not match its payload. Validation is invoked from nextFetchedRecord at CompletedFetch.java:232 for each record returned to the user.

Common situations: Topics still on v0/v1 message format because inter.broker.protocol.version / log.message.format.version are pinned low on the broker; reading historical segments produced before a format upgrade; same hardware-level corruption sources as the v2 batch case; downgrade scenarios after a failed format migration.

Related errors

• Record batch for partition {} at offset {} is invalid, cause
• Encountered corrupt message when fetching offset {} for topi
• Record size is less than the minimum record overhead (%d)
• Record is corrupt (stored crc = {}, computed crc = {})
• Incorrect declared batch size, premature EOF reached

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/39984f5eefd334c0.json. Report an issue: GitHub.