{"id":"ab94cd4f2163f713","repo":"apache/kafka","slug":"invalid-wrapper-magic-found-in-legacy-deep-record","errorCode":null,"errorMessage":"Invalid wrapper magic found in legacy deep record iterator {}","messagePattern":"Invalid wrapper magic found in legacy deep record iterator (.+?)","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"critical","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java","lineNumber":327,"sourceCode":"            batchBuffer.flip();\n\n            return new BasicLegacyRecordBatch(offset, new LegacyRecord(batchBuffer));\n        }\n    }\n\n    private static class DeepRecordsIterator extends AbstractIterator<Record> implements CloseableIterator<Record> {\n        private final ArrayDeque<AbstractLegacyRecordBatch> innerEntries;\n        private final long absoluteBaseOffset;\n        private final byte wrapperMagic;\n\n        private DeepRecordsIterator(AbstractLegacyRecordBatch wrapperEntry,\n                                    boolean ensureMatchingMagic,\n                                    int maxMessageSize,\n                                    BufferSupplier bufferSupplier) {\n            LegacyRecord wrapperRecord = wrapperEntry.outerRecord();\n            this.wrapperMagic = wrapperRecord.magic();\n            if (wrapperMagic != RecordBatch.MAGIC_VALUE_V0 && wrapperMagic != RecordBatch.MAGIC_VALUE_V1)\n                throw new InvalidRecordException(\"Invalid wrapper magic found in legacy deep record iterator \" + wrapperMagic);\n\n            CompressionType compressionType = wrapperRecord.compressionType();\n            if (compressionType == CompressionType.ZSTD)\n                throw new InvalidRecordException(\"Invalid wrapper compressionType found in legacy deep record iterator \" + wrapperMagic);\n            ByteBuffer wrapperValue = wrapperRecord.value();\n            if (wrapperValue == null)\n                throw new InvalidRecordException(\"Found invalid compressed record set with null value (magic = \" +\n                        wrapperMagic + \")\");\n\n            InputStream stream = Compression.of(compressionType).build().wrapForInput(wrapperValue, wrapperRecord.magic(), bufferSupplier);\n            LogInputStream<AbstractLegacyRecordBatch> logStream = new DataLogInputStream(stream, maxMessageSize);\n\n            long lastOffsetFromWrapper = wrapperEntry.lastOffset();\n            long timestampFromWrapper = wrapperRecord.timestamp();\n            this.innerEntries = new ArrayDeque<>();\n\n            // If relative offset is used, we need to decompress the entire message first to compute\n            // the absolute offset. For simplicity and because it's a format that is on its way out, we","sourceCodeStart":309,"sourceCodeEnd":345,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java#L309-L345","documentation":"Thrown by the DeepRecordsIterator constructor (line 327) as an InvalidRecordException when the outer (wrapper) record's magic is neither MAGIC_VALUE_V0 (0) nor MAGIC_VALUE_V1 (1). AbstractLegacyRecordBatch only knows how to decompress magic 0/1 message sets; any other magic value means the batch was misclassified as legacy when it is actually v2 or garbage.","triggerScenarios":"Calling iterator()/streamingIterator() on an AbstractLegacyRecordBatch whose outerRecord().magic() returns something other than 0 or 1. This typically should not happen for batches constructed through normal code paths; it indicates either memory corruption, a deserialization bug, or a hand-built buffer that was wrapped as legacy but contains a v2 record.","commonSituations":"Custom code that wraps a ByteBuffer in ByteBufferLegacyRecordBatch without first checking the magic byte. Forcing a v2 record batch through a legacy code path during migration or in a test fixture. Bit rot in a serialized buffer that flipped the magic byte.","solutions":["Before constructing an AbstractLegacyRecordBatch, dispatch on magic: route v2 batches to DefaultRecordBatch and only v0/v1 to the legacy path.","Inspect the buffer with kafka-dump-log to see what magic byte is actually on disk for the failing segment.","If this comes from a test fixture, regenerate the fixture with the current RecordBatchFactory APIs rather than hand-rolling bytes.","Audit custom (de)serializers for places that assume legacy format without a magic check."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// InvalidRecordException is raised inside DeepRecordsIterator when a compressed\n// legacy wrapper has a magic value other than 0 or 1. Catch at iteration boundary.\nimport org.apache.kafka.common.errors.InvalidRecordException;\n\ntry {\n    for (Record r : legacyBatch) { /* process */ }\n} catch (InvalidRecordException e) {\n    // wrapper magic is neither V0 nor V1 => malformed producer data or cross-format mix.\n    // skip the batch and alert; do not retry the same bytes.\n    log.error(\"Invalid legacy wrapper magic; skipping batch on {}\", topicPartition, e);\n}","preventionTips":["Standardize on the v2 record format on producers so the legacy deep-iterator path is never exercised.","Never mix message-format versions inside a single topic; set broker message.format.version once per topic and keep all producers aligned.","If you migrate a topic between format versions, do a clean cutover (new topic or full re-drive) rather than letting v0/v1/v2 batches coexist.","Treat InvalidRecordException from the deep iterator as unrecoverable for that batch: skip it, advance the offset, and raise an alert.","When consuming from external systems (e.g. legacy archives, MirrorMaker from old clusters), validate the source cluster's format before trusting the bytes."],"tags":["magic-version","legacy-record","record-batch","validation"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}