apache/kafka · error · RuntimeException

Block size value must be between 4 and 7

Error message

Block size value must be between 4 and 7

What it means

Thrown by BD.validate() inside Lz4BlockOutputStream when parsing or constructing an LZ4 frame Block Descriptor (BD) byte whose block-size bits encode a value outside the legal 4-7 range. Per the LZ4 frame spec (v1.5.1) only block sizes 64KB(4), 256KB(5), 1MB(6), 4MB(7) are permitted; the size is computed as 2^(2*n+8). It is a plain RuntimeException (unchecked), raised during decompression when BD.fromByte() decodes a non-conformant header, or when a caller manually constructs BD with an invalid code.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockOutputStream.java:372

            this.blockSizeValue = blockSizeValue;
            this.reserved3 = reserved3;
            validate();
        }

        public static BD fromByte(byte bd) {
            int reserved2 = (bd >>> 0) & 15;
            int blockMaximumSize = (bd >>> 4) & 7;
            int reserved3 = (bd >>> 7) & 1;

            return new BD(reserved2, blockMaximumSize, reserved3);
        }

        private void validate() {
            if (reserved2 != 0) {
                throw new RuntimeException("Reserved2 field must be 0");
            }
            if (blockSizeValue < 4 || blockSizeValue > 7) {
                throw new RuntimeException("Block size value must be between 4 and 7");
            }
            if (reserved3 != 0) {
                throw new RuntimeException("Reserved3 field must be 0");
            }
        }

        // 2^(2n+8)
        public int getBlockMaximumSize() {
            return 1 << ((2 * blockSizeValue) + 8);
        }

        public byte toByte() {
            return (byte) (((reserved2 & 15) << 0) | ((blockSizeValue & 7) << 4) | ((reserved3 & 1) << 7));
        }
    }

}

View on GitHub (pinned to c31c9215e1)

Solutions

1. Inspect the full stack trace: a decompression path (FetchResponse/RecordsIterator) points to corrupt or truncated data from a specific topic-partition; verify that broker and re-fetch that batch/segment.
2. Check producer side: ensure records were written by a Kafka client using LZ4 (standard Kafka LZ4 frame), not a raw lz4-java frame stream, and that producer and client library versions agree.
3. If reproducing locally, validate the offending byte sequence against the LZ4 frame format spec; confirm the BD byte's bits 4-6 decode to 4-7.
4. If constructing BD directly in code, pass only BLOCKSIZE_64KB (4) or another value in [4,7].

Example fix

// before (custom/invalid code)
new Lz4BlockOutputStream.BD(3); // throws: 3 < 4

// after
new Lz4BlockOutputStream.BD(Lz4BlockOutputStream.BLOCKSIZE_64KB); // legal: 4 -> 64KB

// Validate an LZ4 frame's BD (Block Descriptor) byte before decompression.
// Layout (per Lz4BlockOutputStream.BD): bits 0-3 reserved2, bits 4-6 blockSizeValue, bit 7 reserved3.
boolean isValidLz4Bd(byte bd) {
    int reserved2 = bd & 0x0F;
    int blockSizeValue = (bd >>> 4) & 0x07;
    int reserved3 = (bd >>> 7) & 0x01;
    return reserved2 == 0 && blockSizeValue >= 4 && blockSizeValue <= 7 && reserved3 == 0;
}
// Only meaningful if you have raw frame bytes; otherwise rely on try-catch below.

// Thrown as RuntimeException from Lz4BlockInputStream when reading a corrupt/malformed LZ4 frame;
// Compression.wrapForInput wraps it in KafkaException.
try {
    try (InputStream in = compression.wrapForInput(buffer, messageVersion, bufferSupplier)) {
        in.read(sink);
    }
} catch (org.apache.kafka.common.KafkaException ke) {
    if (ke.getCause() != null && ke.getCause().getMessage().contains("Block size value")) {
        // corrupt producer payload / wire data — drop the record, do not retry the same bytes
        log.warn("Malformed LZ4 frame rejected", ke);
    } else {
        throw ke;
    }
}

When it happens

Trigger: Decompressing an LZ4-compressed Kafka record batch whose frame BD byte is corrupt/truncated (e.g. via Lz4BlockInputStream reading damaged bytes from a Kafka partition, a tampered message, or a partial fetch response). Also triggered if application code directly calls new Lz4BlockOutputStream.BD(n) with n outside [4,7]. On the write path Kafka always uses BLOCKSIZE_64KB (4), so a non-4 value here almost always indicates bit-rot on the read path.

Common situations: See trigger scenarios.

Related errors

• Reserved3 field must be 0
• Block checksum mismatch
• lz4 doesn't support given compression level: level
• Stream ended prematurely
• Stream unsupported (invalid magic bytes)

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/fb61f37448c97bd0.json. Report an issue: GitHub.