apache/kafka · critical · InvalidRecordException

Invalid wrapper magic found in legacy deep record iterator {

Error message

Invalid wrapper magic found in legacy deep record iterator {}

What it means

Thrown by the DeepRecordsIterator constructor (line 327) as an InvalidRecordException when the outer (wrapper) record's magic is neither MAGIC_VALUE_V0 (0) nor MAGIC_VALUE_V1 (1). AbstractLegacyRecordBatch only knows how to decompress magic 0/1 message sets; any other magic value means the batch was misclassified as legacy when it is actually v2 or garbage.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java:327

            batchBuffer.flip();

            return new BasicLegacyRecordBatch(offset, new LegacyRecord(batchBuffer));
        }
    }

    private static class DeepRecordsIterator extends AbstractIterator<Record> implements CloseableIterator<Record> {
        private final ArrayDeque<AbstractLegacyRecordBatch> innerEntries;
        private final long absoluteBaseOffset;
        private final byte wrapperMagic;

        private DeepRecordsIterator(AbstractLegacyRecordBatch wrapperEntry,
                                    boolean ensureMatchingMagic,
                                    int maxMessageSize,
                                    BufferSupplier bufferSupplier) {
            LegacyRecord wrapperRecord = wrapperEntry.outerRecord();
            this.wrapperMagic = wrapperRecord.magic();
            if (wrapperMagic != RecordBatch.MAGIC_VALUE_V0 && wrapperMagic != RecordBatch.MAGIC_VALUE_V1)
                throw new InvalidRecordException("Invalid wrapper magic found in legacy deep record iterator " + wrapperMagic);

            CompressionType compressionType = wrapperRecord.compressionType();
            if (compressionType == CompressionType.ZSTD)
                throw new InvalidRecordException("Invalid wrapper compressionType found in legacy deep record iterator " + wrapperMagic);
            ByteBuffer wrapperValue = wrapperRecord.value();
            if (wrapperValue == null)
                throw new InvalidRecordException("Found invalid compressed record set with null value (magic = " +
                        wrapperMagic + ")");

            InputStream stream = Compression.of(compressionType).build().wrapForInput(wrapperValue, wrapperRecord.magic(), bufferSupplier);
            LogInputStream<AbstractLegacyRecordBatch> logStream = new DataLogInputStream(stream, maxMessageSize);

            long lastOffsetFromWrapper = wrapperEntry.lastOffset();
            long timestampFromWrapper = wrapperRecord.timestamp();
            this.innerEntries = new ArrayDeque<>();

            // If relative offset is used, we need to decompress the entire message first to compute
            // the absolute offset. For simplicity and because it's a format that is on its way out, we

View on GitHub (pinned to c31c9215e1)

Solutions

Before constructing an AbstractLegacyRecordBatch, dispatch on magic: route v2 batches to DefaultRecordBatch and only v0/v1 to the legacy path.
Inspect the buffer with kafka-dump-log to see what magic byte is actually on disk for the failing segment.
If this comes from a test fixture, regenerate the fixture with the current RecordBatchFactory APIs rather than hand-rolling bytes.
Audit custom (de)serializers for places that assume legacy format without a magic check.

Defensive patterns

Strategy: try-catch

Try / catch

// InvalidRecordException is raised inside DeepRecordsIterator when a compressed
// legacy wrapper has a magic value other than 0 or 1. Catch at iteration boundary.
import org.apache.kafka.common.errors.InvalidRecordException;

try {
    for (Record r : legacyBatch) { /* process */ }
} catch (InvalidRecordException e) {
    // wrapper magic is neither V0 nor V1 => malformed producer data or cross-format mix.
    // skip the batch and alert; do not retry the same bytes.
    log.error("Invalid legacy wrapper magic; skipping batch on {}", topicPartition, e);
}

Prevention

Standardize on the v2 record format on producers so the legacy deep-iterator path is never exercised.
Never mix message-format versions inside a single topic; set broker message.format.version once per topic and keep all producers aligned.
If you migrate a topic between format versions, do a clean cutover (new topic or full re-drive) rather than letting v0/v1/v2 batches coexist.
Treat InvalidRecordException from the deep iterator as unrecoverable for that batch: skip it, advance the offset, and raise an alert.
When consuming from external systems (e.g. legacy archives, MirrorMaker from old clusters), validate the source cluster's format before trusting the bytes.

When it happens

Trigger: Calling iterator()/streamingIterator() on an AbstractLegacyRecordBatch whose outerRecord().magic() returns something other than 0 or 1. This typically should not happen for batches constructed through normal code paths; it indicates either memory corruption, a deserialization bug, or a hand-built buffer that was wrapped as legacy but contains a v2 record.

Common situations: Custom code that wraps a ByteBuffer in ByteBufferLegacyRecordBatch without first checking the magic byte. Forcing a v2 record batch through a legacy code path during migration or in a test fixture. Bit rot in a serialized buffer that flipped the magic byte.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/ab94cd4f2163f713.json. Report an issue: GitHub.