{"id":"37d7eac228d9952c","repo":"apache/kafka","slug":"invalid-wrapper-compressiontype-found-in-legacy-de","errorCode":null,"errorMessage":"Invalid wrapper compressionType found in legacy deep record iterator {}","messagePattern":"Invalid wrapper compressionType found in legacy deep record iterator (.+?)","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java","lineNumber":331,"sourceCode":"    }\n\n    private static class DeepRecordsIterator extends AbstractIterator<Record> implements CloseableIterator<Record> {\n        private final ArrayDeque<AbstractLegacyRecordBatch> innerEntries;\n        private final long absoluteBaseOffset;\n        private final byte wrapperMagic;\n\n        private DeepRecordsIterator(AbstractLegacyRecordBatch wrapperEntry,\n                                    boolean ensureMatchingMagic,\n                                    int maxMessageSize,\n                                    BufferSupplier bufferSupplier) {\n            LegacyRecord wrapperRecord = wrapperEntry.outerRecord();\n            this.wrapperMagic = wrapperRecord.magic();\n            if (wrapperMagic != RecordBatch.MAGIC_VALUE_V0 && wrapperMagic != RecordBatch.MAGIC_VALUE_V1)\n                throw new InvalidRecordException(\"Invalid wrapper magic found in legacy deep record iterator \" + wrapperMagic);\n\n            CompressionType compressionType = wrapperRecord.compressionType();\n            if (compressionType == CompressionType.ZSTD)\n                throw new InvalidRecordException(\"Invalid wrapper compressionType found in legacy deep record iterator \" + wrapperMagic);\n            ByteBuffer wrapperValue = wrapperRecord.value();\n            if (wrapperValue == null)\n                throw new InvalidRecordException(\"Found invalid compressed record set with null value (magic = \" +\n                        wrapperMagic + \")\");\n\n            InputStream stream = Compression.of(compressionType).build().wrapForInput(wrapperValue, wrapperRecord.magic(), bufferSupplier);\n            LogInputStream<AbstractLegacyRecordBatch> logStream = new DataLogInputStream(stream, maxMessageSize);\n\n            long lastOffsetFromWrapper = wrapperEntry.lastOffset();\n            long timestampFromWrapper = wrapperRecord.timestamp();\n            this.innerEntries = new ArrayDeque<>();\n\n            // If relative offset is used, we need to decompress the entire message first to compute\n            // the absolute offset. For simplicity and because it's a format that is on its way out, we\n            // do the same for message format version 0\n            try {\n                while (true) {\n                    AbstractLegacyRecordBatch innerEntry = logStream.nextBatch();","sourceCodeStart":313,"sourceCodeEnd":349,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java#L313-L349","documentation":"Thrown by DeepRecordsIterator (line 331) as an InvalidRecordException when the wrapper record's compressionType equals ZSTD. ZSTD support (KIP-110) was added only for message format v2; legacy v0/v1 message sets may only use NONE, GZIP, SNAPPY, or LZ4. Encountering ZSTD on a legacy wrapper is invalid by spec.","triggerScenarios":"A v0/v1 message batch whose attributes byte encodes ZSTD. Produced by a buggy custom producer, an interop client that mis-sets the attributes byte, or by corruption that flipped bits into the ZSTD code. The check fires the moment DeepRecordsIterator tries to decompress.","commonSituations":"Cross-language clients (e.g. older librdkafka forks) that set the wrong attributes byte. Bit-flip corruption in the attributes byte of a legacy record. Test code that constructs LegacyRecord with CompressionType.ZSTD directly. A producer forced to magic=1 but configured for zstd compression.","solutions":["Produce with magic v2 if you need ZSTD compression (set message.format.version >= 0.11.0 on the broker).","If you must use legacy v0/v1, restrict producer compression.type to gzip, snappy, lz4, or none.","Reproduce with kafka-dump-log --deep-iteration to confirm which segment/offset has the bad attributes byte and re-send those records correctly.","Audit non-Java producers for the attributes-byte encoding of compression (low 3 bits) to ensure they are not emitting 4 (ZSTD) for v0/v1."],"exampleFix":"// before\nProducerConfig: compression.type=zstd, broker message.format.version=0.10.2 (v1)\n\n// after\nEither set message.format.version=0.11+ (v2) and keep zstd,\nor set compression.type=lz4 while staying on v1.","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Legacy v0/v1 batches must not use ZSTD compression (only v2 supports it).\n// The error surfaces during deep iteration of a compressed legacy wrapper.\nimport org.apache.kafka.common.errors.InvalidRecordException;\n\ntry {\n    for (Record r : legacyBatch) { /* process */ }\n} catch (InvalidRecordException e) {\n    // a v0/v1 producer incorrectly produced ZSTD-compressed records.\n    log.error(\"ZSTD compression not permitted in legacy format; batch is malformed\", e);\n}","preventionTips":["ZSTD is only valid for message format v2 (KIP-110); never enable ZSTD on producers targeting a v0/v1 topic.","Keep compression.type and the topic's message.format.version consistent; if the topic is v0/v1, use NONE/GZIP/SNAPPY/LZ4 only.","When upgrading to ZSTD, raise message.format.version to v2 on the broker and verify producer client version supports KIP-110 before switching.","Quarantine and re-drive batches that trigger this error rather than retrying; the bytes are structurally invalid for the declared format."],"tags":["compression","zstd","magic-v0-v1","producer","validation"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}