{"id":"cb1cd8704f50001f","repo":"apache/kafka","slug":"found-invalid-compressed-record-set-with-null-valu","errorCode":null,"errorMessage":"Found invalid compressed record set with null value (magic = {})","messagePattern":"Found invalid compressed record set with null value \\(magic = (.+?)\\)","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java","lineNumber":334,"sourceCode":"        private final ArrayDeque<AbstractLegacyRecordBatch> innerEntries;\n        private final long absoluteBaseOffset;\n        private final byte wrapperMagic;\n\n        private DeepRecordsIterator(AbstractLegacyRecordBatch wrapperEntry,\n                                    boolean ensureMatchingMagic,\n                                    int maxMessageSize,\n                                    BufferSupplier bufferSupplier) {\n            LegacyRecord wrapperRecord = wrapperEntry.outerRecord();\n            this.wrapperMagic = wrapperRecord.magic();\n            if (wrapperMagic != RecordBatch.MAGIC_VALUE_V0 && wrapperMagic != RecordBatch.MAGIC_VALUE_V1)\n                throw new InvalidRecordException(\"Invalid wrapper magic found in legacy deep record iterator \" + wrapperMagic);\n\n            CompressionType compressionType = wrapperRecord.compressionType();\n            if (compressionType == CompressionType.ZSTD)\n                throw new InvalidRecordException(\"Invalid wrapper compressionType found in legacy deep record iterator \" + wrapperMagic);\n            ByteBuffer wrapperValue = wrapperRecord.value();\n            if (wrapperValue == null)\n                throw new InvalidRecordException(\"Found invalid compressed record set with null value (magic = \" +\n                        wrapperMagic + \")\");\n\n            InputStream stream = Compression.of(compressionType).build().wrapForInput(wrapperValue, wrapperRecord.magic(), bufferSupplier);\n            LogInputStream<AbstractLegacyRecordBatch> logStream = new DataLogInputStream(stream, maxMessageSize);\n\n            long lastOffsetFromWrapper = wrapperEntry.lastOffset();\n            long timestampFromWrapper = wrapperRecord.timestamp();\n            this.innerEntries = new ArrayDeque<>();\n\n            // If relative offset is used, we need to decompress the entire message first to compute\n            // the absolute offset. For simplicity and because it's a format that is on its way out, we\n            // do the same for message format version 0\n            try {\n                while (true) {\n                    AbstractLegacyRecordBatch innerEntry = logStream.nextBatch();\n                    if (innerEntry == null)\n                        break;\n","sourceCodeStart":316,"sourceCodeEnd":352,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java#L316-L352","documentation":"Thrown by DeepRecordsIterator (line 334) as an InvalidRecordException when a compressed wrapper legacy record has a null value. A compressed batch's value field is supposed to contain the compressed inner message-set bytes; a null value means there is nothing to decompress, which is structurally invalid for a compressed message.","triggerScenarios":"Producing or reading a v0/v1 record whose attributes indicate compression but whose value is null. Bubbles up when the broker or consumer calls iterator() on the batch and DeepRecordsIterator tries to wrap the value for decompression at Compression.of(...).wrapForInput(...).","commonSituations":"A custom producer that sets compression on the wrapper but sends a tombstone (null value). Corruption that zeroed the value length. Producer bug where the compressed buffer failed to build and was sent as null. Some versions of non-Java clients mishandling the compressed wrapper for v1.","solutions":["Do not send null values with compression enabled on magic v0/v1; send the actual compressed payload or disable compression.","Upgrade to magic v2, which handles null/tombstone records correctly even inside compressed batches.","If reading existing bad data, isolate the corrupt segment with kafka-dump-log and re-send the affected records through a fixed producer.","Audit custom serializers to ensure they never emit a null value when compression attributes are set."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// A compressed legacy wrapper must carry a non-null payload; null value => invalid.\nimport org.apache.kafka.common.errors.InvalidRecordException;\n\ntry {\n    for (Record r : legacyBatch) { /* process */ }\n} catch (InvalidRecordException e) {\n    log.error(\"Compressed legacy record set has null payload; skipping corrupt batch\", e);\n}","preventionTips":["Never send tombstones (null values) on compressed v0/v1 batches; compression requires a non-null payload to decompress.","When producing tombstones, send them uncompressed or switch to the v2 format which handles them correctly.","Validate producer-side that compression.type is set to NONE when emitting null-valued records on legacy formats.","Treat this as a corrupt batch: skip and advance rather than retry."],"tags":["compression","null-value","magic-v0-v1","producer","validation"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}