apache/kafka · error · InvalidRecordException

Found invalid compressed record set with no inner records

Error message

Found invalid compressed record set with no inner records

What it means

Thrown by DeepRecordsIterator (line 372) as an InvalidRecordException when, after fully decompressing the wrapper value, the innerEntries deque is empty. A compressed message set must contain at least one inner record; an empty compressed payload is invalid (it carries no data and was likely produced by a bug).

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java:372

                    byte magic = record.magic();

                    if (ensureMatchingMagic && magic != wrapperMagic)
                        throw new InvalidRecordException("Compressed message magic " + magic +
                                " does not match wrapper magic " + wrapperMagic);

                    if (magic == RecordBatch.MAGIC_VALUE_V1) {
                        LegacyRecord recordWithTimestamp = new LegacyRecord(
                                record.buffer(),
                                timestampFromWrapper,
                                wrapperRecord.timestampType());
                        innerEntry = new BasicLegacyRecordBatch(innerEntry.lastOffset(), recordWithTimestamp);
                    }

                    innerEntries.addLast(innerEntry);
                }

                if (innerEntries.isEmpty())
                    throw new InvalidRecordException("Found invalid compressed record set with no inner records");

                if (wrapperMagic == RecordBatch.MAGIC_VALUE_V1) {
                    if (lastOffsetFromWrapper == 0) {
                        // The outer offset may be 0 if this is produce data from certain versions of librdkafka.
                        this.absoluteBaseOffset = 0;
                    } else {
                        long lastInnerOffset = innerEntries.getLast().offset();
                        if (lastOffsetFromWrapper < lastInnerOffset)
                            throw new InvalidRecordException("Found invalid wrapper offset in compressed v1 message set, " +
                                    "wrapper offset '" + lastOffsetFromWrapper + "' is less than the last inner message " +
                                    "offset '" + lastInnerOffset + "' and it is not zero.");
                        this.absoluteBaseOffset = lastOffsetFromWrapper - lastInnerOffset;
                    }
                } else {
                    this.absoluteBaseOffset = -1;
                }
            } catch (IOException e) {
                throw new KafkaException(e);

View on GitHub (pinned to c31c9215e1)

Solutions

On the producer, skip sending a batch when the record list is empty (do not produce a compressed wrapper with no inner records).
If reading existing data, identify the bad segment with kafka-dump-log --deep-iteration and re-send the affected partition range from a clean source.
Upgrade the producing client to a current version that does not emit empty compressed batches.
Add a guard in custom producers: if (records.isEmpty()) return; before compressing.

Example fix

// before
try (var producer = new KafkaProducer<>(props)) {
    producer.send(new ProducerRecord<>(topic, compressionEnabledEmptyBatch));
}

// after
if (!records.isEmpty()) {
    producer.send(buildBatch(records));
}

Defensive patterns

Strategy: try-catch

Try / catch

// A compressed legacy message set must contain at least one inner record.
import org.apache.kafka.common.errors.InvalidRecordException;

try {
    for (Record r : legacyBatch) { /* process */ }
} catch (InvalidRecordException e) {
    log.error("Compressed legacy batch decoded to zero inner records; corrupt or malformed", e);
}

Prevention

Avoid producing empty batches with compression enabled; producers should suppress empty compressed batches.
Keep producer client versions current; older buggy producer builds could emit empty compressed message sets.
Treat the batch as corrupt and skip; an empty compressed set is not a transient failure so retry is pointless.
If this recurs for a topic, inspect producer logic that may flush compression buffers with no pending records.

When it happens

Trigger: Producing a compressed batch with zero records, or a corruption/truncation that leaves the compressed stream readable but empty. Triggered on the consumer or broker when iterator() runs over the wrapper and the decompressed LogInputStream yields no batches.

Common situations: A producer that builds a compressed batch from an empty record collection and still sends it. Buffer truncation that drops just the inner bytes while leaving headers intact. A custom serializer returning an empty list under some code path. Pre-0.10 librdkafka had edge cases producing empty compressed wrappers.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/d28615ba8ca77f51.json. Report an issue: GitHub.