{"id":"d28615ba8ca77f51","repo":"apache/kafka","slug":"found-invalid-compressed-record-set-with-no-inner","errorCode":null,"errorMessage":"Found invalid compressed record set with no inner records","messagePattern":"Found invalid compressed record set with no inner records","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java","lineNumber":372,"sourceCode":"                    byte magic = record.magic();\n\n                    if (ensureMatchingMagic && magic != wrapperMagic)\n                        throw new InvalidRecordException(\"Compressed message magic \" + magic +\n                                \" does not match wrapper magic \" + wrapperMagic);\n\n                    if (magic == RecordBatch.MAGIC_VALUE_V1) {\n                        LegacyRecord recordWithTimestamp = new LegacyRecord(\n                                record.buffer(),\n                                timestampFromWrapper,\n                                wrapperRecord.timestampType());\n                        innerEntry = new BasicLegacyRecordBatch(innerEntry.lastOffset(), recordWithTimestamp);\n                    }\n\n                    innerEntries.addLast(innerEntry);\n                }\n\n                if (innerEntries.isEmpty())\n                    throw new InvalidRecordException(\"Found invalid compressed record set with no inner records\");\n\n                if (wrapperMagic == RecordBatch.MAGIC_VALUE_V1) {\n                    if (lastOffsetFromWrapper == 0) {\n                        // The outer offset may be 0 if this is produce data from certain versions of librdkafka.\n                        this.absoluteBaseOffset = 0;\n                    } else {\n                        long lastInnerOffset = innerEntries.getLast().offset();\n                        if (lastOffsetFromWrapper < lastInnerOffset)\n                            throw new InvalidRecordException(\"Found invalid wrapper offset in compressed v1 message set, \" +\n                                    \"wrapper offset '\" + lastOffsetFromWrapper + \"' is less than the last inner message \" +\n                                    \"offset '\" + lastInnerOffset + \"' and it is not zero.\");\n                        this.absoluteBaseOffset = lastOffsetFromWrapper - lastInnerOffset;\n                    }\n                } else {\n                    this.absoluteBaseOffset = -1;\n                }\n            } catch (IOException e) {\n                throw new KafkaException(e);","sourceCodeStart":354,"sourceCodeEnd":390,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java#L354-L390","documentation":"Thrown by DeepRecordsIterator (line 372) as an InvalidRecordException when, after fully decompressing the wrapper value, the innerEntries deque is empty. A compressed message set must contain at least one inner record; an empty compressed payload is invalid (it carries no data and was likely produced by a bug).","triggerScenarios":"Producing a compressed batch with zero records, or a corruption/truncation that leaves the compressed stream readable but empty. Triggered on the consumer or broker when iterator() runs over the wrapper and the decompressed LogInputStream yields no batches.","commonSituations":"A producer that builds a compressed batch from an empty record collection and still sends it. Buffer truncation that drops just the inner bytes while leaving headers intact. A custom serializer returning an empty list under some code path. Pre-0.10 librdkafka had edge cases producing empty compressed wrappers.","solutions":["On the producer, skip sending a batch when the record list is empty (do not produce a compressed wrapper with no inner records).","If reading existing data, identify the bad segment with kafka-dump-log --deep-iteration and re-send the affected partition range from a clean source.","Upgrade the producing client to a current version that does not emit empty compressed batches.","Add a guard in custom producers: if (records.isEmpty()) return; before compressing."],"exampleFix":"// before\ntry (var producer = new KafkaProducer<>(props)) {\n    producer.send(new ProducerRecord<>(topic, compressionEnabledEmptyBatch));\n}\n\n// after\nif (!records.isEmpty()) {\n    producer.send(buildBatch(records));\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// A compressed legacy message set must contain at least one inner record.\nimport org.apache.kafka.common.errors.InvalidRecordException;\n\ntry {\n    for (Record r : legacyBatch) { /* process */ }\n} catch (InvalidRecordException e) {\n    log.error(\"Compressed legacy batch decoded to zero inner records; corrupt or malformed\", e);\n}","preventionTips":["Avoid producing empty batches with compression enabled; producers should suppress empty compressed batches.","Keep producer client versions current; older buggy producer builds could emit empty compressed message sets.","Treat the batch as corrupt and skip; an empty compressed set is not a transient failure so retry is pointless.","If this recurs for a topic, inspect producer logic that may flush compression buffers with no pending records."],"tags":["compression","empty-batch","legacy-record","producer","validation"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}