apache/kafka · error · InvalidRecordException

Compressed message magic {} does not match wrapper magic {}

Error message

Compressed message magic {} does not match wrapper magic {}

What it means

Thrown by DeepRecordsIterator (line 357) as an InvalidRecordException when ensureMatchingMagic is true and an inner record's magic differs from the wrapper magic. A compressed v0/v1 message set must be homogeneous: every inner record must share the wrapper's magic version. Mixing magic versions inside one compressed batch is illegal.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java:357

            long lastOffsetFromWrapper = wrapperEntry.lastOffset();
            long timestampFromWrapper = wrapperRecord.timestamp();
            this.innerEntries = new ArrayDeque<>();

            // If relative offset is used, we need to decompress the entire message first to compute
            // the absolute offset. For simplicity and because it's a format that is on its way out, we
            // do the same for message format version 0
            try {
                while (true) {
                    AbstractLegacyRecordBatch innerEntry = logStream.nextBatch();
                    if (innerEntry == null)
                        break;

                    LegacyRecord record = innerEntry.outerRecord();
                    byte magic = record.magic();

                    if (ensureMatchingMagic && magic != wrapperMagic)
                        throw new InvalidRecordException("Compressed message magic " + magic +
                                " does not match wrapper magic " + wrapperMagic);

                    if (magic == RecordBatch.MAGIC_VALUE_V1) {
                        LegacyRecord recordWithTimestamp = new LegacyRecord(
                                record.buffer(),
                                timestampFromWrapper,
                                wrapperRecord.timestampType());
                        innerEntry = new BasicLegacyRecordBatch(innerEntry.lastOffset(), recordWithTimestamp);
                    }

                    innerEntries.addLast(innerEntry);
                }

                if (innerEntries.isEmpty())
                    throw new InvalidRecordException("Found invalid compressed record set with no inner records");

                if (wrapperMagic == RecordBatch.MAGIC_VALUE_V1) {
                    if (lastOffsetFromWrapper == 0) {

View on GitHub (pinned to c31c9215e1)

Solutions

Ensure the producer writes inner records with the same magic as the wrapper (single message.format.version for the whole batch).
After a broker message.format.version change, re-produce the data rather than letting mixed-magic batches linger.
If the data is historical, consume it with a client path that does not enforce matching magic (the public AbstractLegacyRecordBatch.iterator() passes false), then re-produce as v2.
Audit custom (de)serializers and MirrorMaker pipelines for places that rewrap compressed batches without normalizing inner magic.

Defensive patterns

Strategy: try-catch

Try / catch

// Inner records inside a compressed legacy wrapper must use the same magic as the
// wrapper (enforced when ensureMatchingMagic=true, the default for produce paths).
import org.apache.kafka.common.errors.InvalidRecordException;

try {
    for (Record r : legacyBatch) { /* process */ }
} catch (InvalidRecordException e) {
    // inner magic differs from wrapper magic => mixed-format produce or corrupt data.
    log.error("Legacy compressed batch has mismatched inner/wrapper magic", e);
}

Prevention

Ensure all records inside one compressed batch are produced with the same client/format version; do not interleave v0 and v1 producers into one batch.
Pin compression.batch.size and the producer's message format so a single batch never spans format upgrades.
When migrating formats, drain and recreate batches rather than letting old and new magic coexist in a compressed wrapper.
Treat the mismatch as unrecoverable corruption for that batch; skip and alert.

When it happens

Trigger: iterator() on a compressed legacy batch where the decompressed inner records were written with a different magic than the outer wrapper. The check is only enforced when ensureMatchingMagic=true (the public iterator() path passes false, but internal broker validation and several call sites pass true).

Common situations: A producer was upgraded mid-stream and wrote mixed-magic inner records into a single compressed batch. A custom client that reuses a buffer with stale inner records after a magic bump. Corruption that flipped a magic byte in the inner stream. MirrorMaker / replication across clusters with different message.format.version where the wrapper was rewritten but inner records were not.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/09157f9419be3334.json. Report an issue: GitHub.