{"id":"906dc35d2439fab8","repo":"apache/kafka","slug":"inner-messages-must-not-be-compressed","errorCode":null,"errorMessage":"Inner messages must not be compressed","messagePattern":"Inner messages must not be compressed","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java","lineNumber":410,"sourceCode":"                Utils.closeQuietly(stream, \"records iterator stream\");\n            }\n        }\n\n        @Override\n        protected Record makeNext() {\n            if (innerEntries.isEmpty())\n                return allDone();\n\n            AbstractLegacyRecordBatch entry = innerEntries.remove();\n\n            // Convert offset to absolute offset if needed.\n            if (wrapperMagic == RecordBatch.MAGIC_VALUE_V1) {\n                long absoluteOffset = absoluteBaseOffset + entry.offset();\n                entry = new BasicLegacyRecordBatch(absoluteOffset, entry.outerRecord());\n            }\n\n            if (entry.isCompressed())\n                throw new InvalidRecordException(\"Inner messages must not be compressed\");\n\n            return entry;\n        }\n\n        @Override\n        public void close() {}\n    }\n\n    private static class BasicLegacyRecordBatch extends AbstractLegacyRecordBatch {\n        private final LegacyRecord record;\n        private final long offset;\n\n        private BasicLegacyRecordBatch(long offset, LegacyRecord record) {\n            this.offset = offset;\n            this.record = record;\n        }\n\n        @Override","sourceCodeStart":392,"sourceCodeEnd":428,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java#L392-L428","documentation":"Thrown by DeepRecordsIterator.makeNext() (line 410) as an InvalidRecordException when an inner entry returned by the decompressed stream is itself compressed. The legacy format allows compression only at the wrapper level; nested compression (a compressed record inside a compressed wrapper) is invalid and was never supported. The check fires lazily as the iterator yields each inner record.","triggerScenarios":"Iterating a compressed legacy batch whose inner stream contains a record whose attributes byte marks it as compressed. Produced by a buggy client that compresses inner records and then compresses the wrapper again, or by corruption that set compression bits on an inner record. The exception is raised on the first call to next()/hasNext() that hits such an inner entry.","commonSituations":"A custom producer that applies compression twice (inner and outer). A non-Java client with a serialization bug that flips compression bits on inner records. Test fixtures built by manually setting attributes bytes. Corruption in the attributes byte of an inner record.","solutions":["Compress only the wrapper, not the inner records: build inner LegacyRecords with CompressionType.NONE and let the producer compress the outer batch.","Migrate to magic v2, whose RecordBatch format has a single compression flag at the batch level and cannot be nested.","Reproduce with kafka-dump-log --deep-iteration to find the offending offset and re-send those records correctly.","Audit custom and non-Java producers for any code path that sets compression attributes on inner records."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Nested compression is illegal: an inner record of a decompressed legacy batch\n// must itself be uncompressed.\nimport org.apache.kafka.common.errors.InvalidRecordException;\n\ntry {\n    for (Record r : legacyBatch) { /* process */ }\n} catch (InvalidRecordException e) {\n    log.error(\"Legacy batch contains a nested-compressed inner record; malformed producer data\", e);\n}","preventionTips":["Produce only one layer of compression: the outer batch is compressed, inner records are not. Do not wrap an already-compressed message set inside another compressed batch.","Do not manually recompress records that came out of another compressed batch; pass them through uncompressed or re-batch them as a fresh single layer.","When piping data between topics/formats, disable compression on the source read and apply it only once at the target producer.","Treat the batch as corrupt; nested compression is structurally invalid and will not succeed on retry."],"tags":["compression","nested-compression","legacy-record","producer","validation"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}