{"id":"09157f9419be3334","repo":"apache/kafka","slug":"compressed-message-magic-does-not-match-wrapper","errorCode":null,"errorMessage":"Compressed message magic {} does not match wrapper magic {}","messagePattern":"Compressed message magic (.+?) does not match wrapper magic (.+?)","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java","lineNumber":357,"sourceCode":"\n            long lastOffsetFromWrapper = wrapperEntry.lastOffset();\n            long timestampFromWrapper = wrapperRecord.timestamp();\n            this.innerEntries = new ArrayDeque<>();\n\n            // If relative offset is used, we need to decompress the entire message first to compute\n            // the absolute offset. For simplicity and because it's a format that is on its way out, we\n            // do the same for message format version 0\n            try {\n                while (true) {\n                    AbstractLegacyRecordBatch innerEntry = logStream.nextBatch();\n                    if (innerEntry == null)\n                        break;\n\n                    LegacyRecord record = innerEntry.outerRecord();\n                    byte magic = record.magic();\n\n                    if (ensureMatchingMagic && magic != wrapperMagic)\n                        throw new InvalidRecordException(\"Compressed message magic \" + magic +\n                                \" does not match wrapper magic \" + wrapperMagic);\n\n                    if (magic == RecordBatch.MAGIC_VALUE_V1) {\n                        LegacyRecord recordWithTimestamp = new LegacyRecord(\n                                record.buffer(),\n                                timestampFromWrapper,\n                                wrapperRecord.timestampType());\n                        innerEntry = new BasicLegacyRecordBatch(innerEntry.lastOffset(), recordWithTimestamp);\n                    }\n\n                    innerEntries.addLast(innerEntry);\n                }\n\n                if (innerEntries.isEmpty())\n                    throw new InvalidRecordException(\"Found invalid compressed record set with no inner records\");\n\n                if (wrapperMagic == RecordBatch.MAGIC_VALUE_V1) {\n                    if (lastOffsetFromWrapper == 0) {","sourceCodeStart":339,"sourceCodeEnd":375,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java#L339-L375","documentation":"Thrown by DeepRecordsIterator (line 357) as an InvalidRecordException when ensureMatchingMagic is true and an inner record's magic differs from the wrapper magic. A compressed v0/v1 message set must be homogeneous: every inner record must share the wrapper's magic version. Mixing magic versions inside one compressed batch is illegal.","triggerScenarios":"iterator() on a compressed legacy batch where the decompressed inner records were written with a different magic than the outer wrapper. The check is only enforced when ensureMatchingMagic=true (the public iterator() path passes false, but internal broker validation and several call sites pass true).","commonSituations":"A producer was upgraded mid-stream and wrote mixed-magic inner records into a single compressed batch. A custom client that reuses a buffer with stale inner records after a magic bump. Corruption that flipped a magic byte in the inner stream. MirrorMaker / replication across clusters with different message.format.version where the wrapper was rewritten but inner records were not.","solutions":["Ensure the producer writes inner records with the same magic as the wrapper (single message.format.version for the whole batch).","After a broker message.format.version change, re-produce the data rather than letting mixed-magic batches linger.","If the data is historical, consume it with a client path that does not enforce matching magic (the public AbstractLegacyRecordBatch.iterator() passes false), then re-produce as v2.","Audit custom (de)serializers and MirrorMaker pipelines for places that rewrap compressed batches without normalizing inner magic."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Inner records inside a compressed legacy wrapper must use the same magic as the\n// wrapper (enforced when ensureMatchingMagic=true, the default for produce paths).\nimport org.apache.kafka.common.errors.InvalidRecordException;\n\ntry {\n    for (Record r : legacyBatch) { /* process */ }\n} catch (InvalidRecordException e) {\n    // inner magic differs from wrapper magic => mixed-format produce or corrupt data.\n    log.error(\"Legacy compressed batch has mismatched inner/wrapper magic\", e);\n}","preventionTips":["Ensure all records inside one compressed batch are produced with the same client/format version; do not interleave v0 and v1 producers into one batch.","Pin compression.batch.size and the producer's message format so a single batch never spans format upgrades.","When migrating formats, drain and recreate batches rather than letting old and new magic coexist in a compressed wrapper.","Treat the mismatch as unrecoverable corruption for that batch; skip and alert."],"tags":["magic-version","compression","legacy-record","producer","validation"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}