apache/kafka · error · InvalidRecordException

Found invalid compressed record set with null value (magic =

Error message

Found invalid compressed record set with null value (magic = {})

What it means

Thrown by DeepRecordsIterator (line 334) as an InvalidRecordException when a compressed wrapper legacy record has a null value. A compressed batch's value field is supposed to contain the compressed inner message-set bytes; a null value means there is nothing to decompress, which is structurally invalid for a compressed message.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java:334

        private final ArrayDeque<AbstractLegacyRecordBatch> innerEntries;
        private final long absoluteBaseOffset;
        private final byte wrapperMagic;

        private DeepRecordsIterator(AbstractLegacyRecordBatch wrapperEntry,
                                    boolean ensureMatchingMagic,
                                    int maxMessageSize,
                                    BufferSupplier bufferSupplier) {
            LegacyRecord wrapperRecord = wrapperEntry.outerRecord();
            this.wrapperMagic = wrapperRecord.magic();
            if (wrapperMagic != RecordBatch.MAGIC_VALUE_V0 && wrapperMagic != RecordBatch.MAGIC_VALUE_V1)
                throw new InvalidRecordException("Invalid wrapper magic found in legacy deep record iterator " + wrapperMagic);

            CompressionType compressionType = wrapperRecord.compressionType();
            if (compressionType == CompressionType.ZSTD)
                throw new InvalidRecordException("Invalid wrapper compressionType found in legacy deep record iterator " + wrapperMagic);
            ByteBuffer wrapperValue = wrapperRecord.value();
            if (wrapperValue == null)
                throw new InvalidRecordException("Found invalid compressed record set with null value (magic = " +
                        wrapperMagic + ")");

            InputStream stream = Compression.of(compressionType).build().wrapForInput(wrapperValue, wrapperRecord.magic(), bufferSupplier);
            LogInputStream<AbstractLegacyRecordBatch> logStream = new DataLogInputStream(stream, maxMessageSize);

            long lastOffsetFromWrapper = wrapperEntry.lastOffset();
            long timestampFromWrapper = wrapperRecord.timestamp();
            this.innerEntries = new ArrayDeque<>();

            // If relative offset is used, we need to decompress the entire message first to compute
            // the absolute offset. For simplicity and because it's a format that is on its way out, we
            // do the same for message format version 0
            try {
                while (true) {
                    AbstractLegacyRecordBatch innerEntry = logStream.nextBatch();
                    if (innerEntry == null)
                        break;

View on GitHub (pinned to c31c9215e1)

Solutions

Do not send null values with compression enabled on magic v0/v1; send the actual compressed payload or disable compression.
Upgrade to magic v2, which handles null/tombstone records correctly even inside compressed batches.
If reading existing bad data, isolate the corrupt segment with kafka-dump-log and re-send the affected records through a fixed producer.
Audit custom serializers to ensure they never emit a null value when compression attributes are set.

Defensive patterns

Strategy: try-catch

Try / catch

// A compressed legacy wrapper must carry a non-null payload; null value => invalid.
import org.apache.kafka.common.errors.InvalidRecordException;

try {
    for (Record r : legacyBatch) { /* process */ }
} catch (InvalidRecordException e) {
    log.error("Compressed legacy record set has null payload; skipping corrupt batch", e);
}

Prevention

Never send tombstones (null values) on compressed v0/v1 batches; compression requires a non-null payload to decompress.
When producing tombstones, send them uncompressed or switch to the v2 format which handles them correctly.
Validate producer-side that compression.type is set to NONE when emitting null-valued records on legacy formats.
Treat this as a corrupt batch: skip and advance rather than retry.

When it happens

Trigger: Producing or reading a v0/v1 record whose attributes indicate compression but whose value is null. Bubbles up when the broker or consumer calls iterator() on the batch and DeepRecordsIterator tries to wrap the value for decompression at Compression.of(...).wrapForInput(...).

Common situations: A custom producer that sets compression on the wrapper but sends a tombstone (null value). Corruption that zeroed the value length. Producer bug where the compressed buffer failed to build and was sent as null. Some versions of non-Java clients mishandling the compressed wrapper for v1.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/cb1cd8704f50001f.json. Report an issue: GitHub.