What causes "Inner messages must not be compressed" in apache/kafka?

Compress only the wrapper, not the inner records: build inner LegacyRecords with CompressionType.NONE and let the producer compress the outer batch.

apache/kafka · error · InvalidRecordException

Inner messages must not be compressed

Error message

Inner messages must not be compressed

What it means

Thrown by DeepRecordsIterator.makeNext() (line 410) as an InvalidRecordException when an inner entry returned by the decompressed stream is itself compressed. The legacy format allows compression only at the wrapper level; nested compression (a compressed record inside a compressed wrapper) is invalid and was never supported. The check fires lazily as the iterator yields each inner record.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java:410

                Utils.closeQuietly(stream, "records iterator stream");
            }
        }

        @Override
        protected Record makeNext() {
            if (innerEntries.isEmpty())
                return allDone();

            AbstractLegacyRecordBatch entry = innerEntries.remove();

            // Convert offset to absolute offset if needed.
            if (wrapperMagic == RecordBatch.MAGIC_VALUE_V1) {
                long absoluteOffset = absoluteBaseOffset + entry.offset();
                entry = new BasicLegacyRecordBatch(absoluteOffset, entry.outerRecord());
            }

            if (entry.isCompressed())
                throw new InvalidRecordException("Inner messages must not be compressed");

            return entry;
        }

        @Override
        public void close() {}
    }

    private static class BasicLegacyRecordBatch extends AbstractLegacyRecordBatch {
        private final LegacyRecord record;
        private final long offset;

        private BasicLegacyRecordBatch(long offset, LegacyRecord record) {
            this.offset = offset;
            this.record = record;
        }

        @Override

View on GitHub (pinned to c31c9215e1)

Solutions

Compress only the wrapper, not the inner records: build inner LegacyRecords with CompressionType.NONE and let the producer compress the outer batch.
Migrate to magic v2, whose RecordBatch format has a single compression flag at the batch level and cannot be nested.
Reproduce with kafka-dump-log --deep-iteration to find the offending offset and re-send those records correctly.
Audit custom and non-Java producers for any code path that sets compression attributes on inner records.

Defensive patterns

Strategy: try-catch

Try / catch

// Nested compression is illegal: an inner record of a decompressed legacy batch
// must itself be uncompressed.
import org.apache.kafka.common.errors.InvalidRecordException;

try {
    for (Record r : legacyBatch) { /* process */ }
} catch (InvalidRecordException e) {
    log.error("Legacy batch contains a nested-compressed inner record; malformed producer data", e);
}

Prevention

Produce only one layer of compression: the outer batch is compressed, inner records are not. Do not wrap an already-compressed message set inside another compressed batch.
Do not manually recompress records that came out of another compressed batch; pass them through uncompressed or re-batch them as a fresh single layer.
When piping data between topics/formats, disable compression on the source read and apply it only once at the target producer.
Treat the batch as corrupt; nested compression is structurally invalid and will not succeed on retry.

When it happens

Trigger: Iterating a compressed legacy batch whose inner stream contains a record whose attributes byte marks it as compressed. Produced by a buggy client that compresses inner records and then compresses the wrapper again, or by corruption that set compression bits on an inner record. The exception is raised on the first call to next()/hasNext() that hits such an inner entry.

Common situations: A custom producer that applies compression twice (inner and outer). A non-Java client with a serialization bug that flips compression bits on inner records. Test fixtures built by manually setting attributes bytes. Corruption in the attributes byte of an inner record.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/906dc35d2439fab8.json. Report an issue: GitHub.