apache/kafka · error · RuntimeException

Dependent block stream is unsupported

Error message

Dependent block stream is unsupported

What it means

Thrown by FLG.validate() when the Block-Independence bit (bit 5 of FLG) is 0, meaning the producer requested a 'dependent block stream' where each block reuses the previous block's dictionary. Kafka's Lz4BlockInputStream/OutputStream only implement independent-block mode (the LZ4 frame default), so a dependent-block frame is rejected.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockOutputStream.java:318

            return new FLG(reserved,
                           contentChecksum,
                           contentSize,
                           blockChecksum,
                           blockIndependence,
                           version);
        }

        public byte toByte() {
            return (byte) (((reserved & 3) << 0) | ((contentChecksum & 1) << 2)
                    | ((contentSize & 1) << 3) | ((blockChecksum & 1) << 4) | ((blockIndependence & 1) << 5) | ((version & 3) << 6));
        }

        private void validate() {
            if (reserved != 0) {
                throw new RuntimeException("Reserved bits must be 0");
            }
            if (blockIndependence != 1) {
                throw new RuntimeException("Dependent block stream is unsupported");
            }
            if (version != VERSION) {
                throw new RuntimeException(String.format("Version %d is unsupported", version));
            }
        }

        public boolean isContentChecksumSet() {
            return contentChecksum == 1;
        }

        public boolean isContentSizeSet() {
            return contentSize == 1;
        }

        public boolean isBlockChecksumSet() {
            return blockChecksum == 1;
        }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Re-compress the data using Kafka's CompressionType.LZ4 (or any producer emitting blockIndependence=1).
2. If you must consume external LZ4 data, decompress it with a general-purpose LZ4 library that supports linked-block mode before it reaches Kafka's Lz4BlockInputStream.
3. Inspect the FLG byte: bit 5 must be 1 for Kafka compatibility.

Example fix

// before: external lz4 CLI wrote linked-block frame
//   lz4 --block-dependency file.dat file.dat.lz4
//   -> consumed via Kafka custom serializer -> throws

// after: produce the frame in independent-block mode
//   lz4 file.dat file.dat.lz4    (default is independent)
// or in code, use Kafka's own compressor so blockIndependence is always 1:

// FLG.blockIndependence is bit 5; Kafka only emits block-independence=1.
byte flgByte = frameBuf.get(flgOffset);
if ((flgByte & 0x20) == 0) {
    throw new IOException("Refusing LZ4 frame: dependent-block mode is not supported by Kafka");
}

null

try {
    try (Lz4BlockInputStream in = new Lz4BlockInputStream(payload, BufferSupplier.NO_CACHING, false)) {
        // consume
    }
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().contains("Dependent block stream is unsupported")) {
        // Producer used a non-Kafka LZ4 encoder in dependent-block mode; reject the record.
        log.warn("Rejecting LZ4 frame produced in dependent-block mode on {}", tp);
        return;
    }
    throw e;
}

When it happens

Trigger: FLG.fromByte(b) where bit 5 == 0, typically while reading a frame produced by an LZ4 tool configured for linked/block-dependent mode. Reached from readHeader() at line 127. The Kafka producer always writes blockIndependence=1 (see FLG public constructor line 274).

Common situations: Reading Kafka records compressed by a non-Kafka LZ4 producer (e.g. lz4 CLI with --block-dependency or a streaming library that defaults to linked blocks); ingesting external LZ4 data through a custom Kafka serializer; consuming data that was re-compressed by an intermediary using block-dependent mode.

Related errors

• Reserved bits must be 0
• Version %d is unsupported
• Reserved2 field must be 0
• Block checksum mismatch
• mark not supported

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/ccaa75fa6530ac73.json. Report an issue: GitHub.