{"id":"167b8cf040a62537","repo":"apache/kafka","slug":"stream-frame-descriptor-corrupted","errorCode":null,"errorMessage":"Stream frame descriptor corrupted","messagePattern":"Stream frame descriptor corrupted","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java","lineNumber":150,"sourceCode":"                throw new IOException(PREMATURE_EOS);\n            }\n            in.position(in.position() + 8);\n        }\n\n        // Final byte of Frame Descriptor is HC checksum\n\n        // Old implementations produced incorrect HC checksums\n        if (ignoreFlagDescriptorChecksum) {\n            in.position(in.position() + 1);\n            return;\n        }\n\n        int len = in.position() - in.reset().position();\n\n        int hash = CHECKSUM.hash(in, in.position(), len, 0);\n        in.position(in.position() + len);\n        if (in.get() != (byte) ((hash >> 8) & 0xFF)) {\n            throw new IOException(DESCRIPTOR_HASH_MISMATCH);\n        }\n    }\n\n    /**\n     * Decompresses (if necessary) buffered data, optionally computes and validates a XXHash32 checksum, and writes the\n     * result to a buffer.\n     *\n     * @throws IOException\n     */\n    private void readBlock() throws IOException {\n        if (in.remaining() < 4) {\n            throw new IOException(PREMATURE_EOS);\n        }\n\n        int blockSize = in.getInt();\n        boolean compressed = (blockSize & LZ4_FRAME_INCOMPRESSIBLE_MASK) == 0;\n        blockSize &= ~LZ4_FRAME_INCOMPRESSIBLE_MASK;\n","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockInputStream.java#L132-L168","documentation":"Thrown in readHeader() when the HC (Header Checksum) byte computed over the FLG/BD descriptor (and optional content size) does not equal the byte stored in the frame. LZ4 frames include a one-byte XXHash32-derived checksum of the frame descriptor; a mismatch means the descriptor bytes are corrupt or were produced by an implementation with an incorrect checksum calculation. Raised as IOException(DESCRIPTOR_HASH_MISMATCH) unless ignoreFlagDescriptorChecksum was set to bypass it.","triggerScenarios":"Decompressing an LZ4 frame whose descriptor bytes were corrupted, truncated-and-rewritten, or produced by old/buggy Kafka clients (<0.10) that computed the HC byte incorrectly, while the consumer was constructed with ignoreFlagDescriptorChecksum=false. Also triggered by non-conforming third-party LZ4 encoders.","commonSituations":"Interoperating with very old Kafka clients whose LZ4 HC calculation was wrong; bit-rot/corruption on disk; memory/network bit flips; hand-edited frames; consumer constructed without the legacy-compat flag when reading legacy data.","solutions":["If reading data written by old Kafka clients, construct Lz4BlockInputStream with ignoreFlagDescriptorChecksum=true (the path Kafka uses for legacy batches) to bypass the HC check.","If data is from a conformant producer, investigate corruption: disk, network, buffer reuse, or concurrent modification.","Re-encode the data with a current Kafka client using Lz4BlockOutputStream to get a correct descriptor checksum.","Confirm the LZ4 encoder (if non-Kafka) implements the v1.5.1 frame HC byte correctly."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Optional: verify descriptor checksum yourself only if interoperating with hand-rolled LZ4 producers.\n// Normally, do NOT replicate the XXHash32 check; rely on the stream to throw.\n// If interoperating with legacy producers known to emit bad HC, construct with ignoreFlagDescriptorChecksum=true:\nnew Lz4BlockInputStream(buffer, /* ignoreFlagDescriptorChecksum */ true);","typeGuard":null,"tryCatchPattern":"try {\n    try (Lz4BlockInputStream in = new Lz4BlockInputStream(buffer, ignoreFlagDescriptorChecksum)) {\n        // ... read ...\n    }\n} catch (IOException e) {\n    // \"Stream frame descriptor corrupted\" -- HC checksum mismatch;\n    // do NOT silently set ignoreFlagDescriptorChecksum=true to suppress; investigate the producer\n}","preventionTips":["Do NOT default ignoreFlagDescriptorChecksum=true to mask the failure; it exists only for legacy producer compatibility.","A genuine HC mismatch usually means bit-flips or a non-conformant producer; investigate root cause.","Pin producer and consumer LZ4 implementations to compatible versions; mismatched XXHash32 variants cause spurious failures.","Capture the offending bytes for offline lz4 --decompress reproduction when diagnosing."],"tags":["compression","lz4","checksum","corruption","io","java"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}