{"id":"96ebe803c437bbdb","repo":"apache/kafka","slug":"block-size-d-exceeded-max-d","errorCode":null,"errorMessage":"Block size %d exceeded max: %d","messagePattern":"Block size (.+?) exceeded max: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java","lineNumber":176,"sourceCode":"     * @throws IOException\n     */\n    private void readBlock() throws IOException {\n        if (in.remaining() < 4) {\n            throw new IOException(PREMATURE_EOS);\n        }\n\n        int blockSize = in.getInt();\n        boolean compressed = (blockSize & LZ4_FRAME_INCOMPRESSIBLE_MASK) == 0;\n        blockSize &= ~LZ4_FRAME_INCOMPRESSIBLE_MASK;\n\n        // Check for EndMark\n        if (blockSize == 0) {\n            finished = true;\n            if (flg.isContentChecksumSet())\n                in.getInt(); // TODO: verify this content checksum\n            return;\n        } else if (blockSize > maxBlockSize) {\n            throw new IOException(String.format(\"Block size %d exceeded max: %d\", blockSize, maxBlockSize));\n        }\n\n        if (in.remaining() < blockSize) {\n            throw new IOException(PREMATURE_EOS);\n        }\n\n        if (compressed) {\n            try {\n                final int bufferSize = DECOMPRESSOR.decompress(in, in.position(), blockSize, decompressionBuffer, 0,\n                    maxBlockSize);\n                decompressionBuffer.position(0);\n                decompressionBuffer.limit(bufferSize);\n                decompressedBuffer = decompressionBuffer;\n            } catch (LZ4Exception e) {\n                throw new IOException(e);\n            }\n        } else {\n            decompressedBuffer = in.slice();","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java#L158-L194","documentation":"Thrown in readBlock() when the decoded block size (after masking off the LZ4_FRAME_INCOMPRESSIBLE_MASK) exceeds the maxBlockSize declared in the frame descriptor (BD byte, one of 64KB/256KB/1MB/4MB). The decompressor pre-allocates a buffer of maxBlockSize and refuses to decompress larger blocks to avoid unbounded memory allocation and to honor the frame contract. Raised as IOException with the offending and max sizes formatted in.","triggerScenarios":"Decompressing an LZ4 frame whose block-size field in a block header is larger than the maxBlockSize set in the BD descriptor byte; produced by a corrupt frame, a non-conformant encoder that wrote oversized blocks, or bit-rot that altered either the BD byte or the block-size field. Reached on read()/skip() when available()==0.","commonSituations":"Corrupt log segments (bit-rot altering size fields); third-party LZ4 encoders ignoring the declared max block size; buffer aliasing where bytes are overwritten; producer/broker memory corruption. Rare in normal Kafka operation since Lz4BlockOutputStream always honors its declared max.","solutions":["Inspect the source data: the BD descriptor declares a max block size; verify the block size field is consistent and not corrupted.","Re-fetch the affected records from a healthy replica to rule out on-disk corruption.","If interop with non-Kafka LZ4 encoders, ensure they emit blocks no larger than the BD-declared maximum (typically 4MB max).","If you control the producer, ensure it uses Kafka's Lz4BlockOutputStream so blocks stay within the declared limit."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Optional pre-check: the declared block max size in the BD byte must be one of\n// {4KiB,64KiB,256KiB,1MiB,4MiB}. A declared block larger than that is already invalid,\n// and any block whose size exceeds the BD-declared max will throw.\n// In practice, do not reimplement frame parsing -- rely on the stream and catch IOException.","typeGuard":null,"tryCatchPattern":"try {\n    try (Lz4BlockInputStream in = new Lz4BlockInputStream(buffer, ignoreFlagDescriptorChecksum)) {\n        // ... read ...\n    }\n} catch (IOException e) {\n    // \"Block size N exceeded max: M\" -- frame declares a smaller max block than the block it carries;\n    // almost always indicates corruption or a hostile/crafted payload\n}","preventionTips":["A block larger than the frame-declared maximum is a strong corruption/tampering signal; never retry the same payload.","Do not raise maxBlockSize to suppress; the limit is a safety bound on decompression buffer allocation.","If legitimately exchanging large LZ4 blocks, configure the producer to declare the correct BD max size (e.g. 4MiB).","Combine with message.size.max limits at the broker to reject oversized payloads early."],"tags":["compression","lz4","corruption","memory","io","java"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}