{"record":{"id":"e3ed75d6e9c6db1b","repo":"apache/hadoop","slug":"writechunk-checksum-size-is-supposed-to-be-bu","errorCode":null,"errorMessage":"writeChunk() checksum size is supposed to be {} but found to be {}","messagePattern":"writeChunk\\(\\) checksum size is supposed to be (.+?) but found to be (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":489,"sourceCode":"    // If packet is full, enqueue it for transmission\n    if (currentPacket.getNumChunks() == currentPacket.getMaxChunks() ||\n            getStreamer().getBytesCurBlock() == blockSize) {\n      enqueueCurrentPacketFull();\n    }\n  }\n\n  private synchronized void writeChunkPrepare(int buflen,\n      int ckoff, int cklen) throws IOException {\n    dfsClient.checkOpen();\n    checkClosed();\n\n    if (buflen > bytesPerChecksum) {\n      throw new IOException(\"writeChunk() buffer size is \" + buflen +\n                            \" is larger than supported  bytesPerChecksum \" +\n                            bytesPerChecksum);\n    }\n    if (cklen != 0 && cklen != getChecksumSize()) {\n      throw new IOException(\"writeChunk() checksum size is supposed to be \" +\n                            getChecksumSize() + \" but found to be \" + cklen);\n    }\n\n    if (currentPacket == null) {\n      currentPacket = createPacket(packetSize, chunksPerPacket, getStreamer()\n          .getBytesCurBlock(), getStreamer().getAndIncCurrentSeqno(), false);\n      DFSClient.LOG.debug(\"WriteChunk allocating new packet seqno={},\"\n              + \" src={}, packetSize={}, chunksPerPacket={}, bytesCurBlock={},\"\n              + \" output stream={}\",\n          currentPacket.getSeqno(), src, packetSize, chunksPerPacket,\n          getStreamer().getBytesCurBlock(), this);\n    }\n  }\n\n  void enqueueCurrentPacket() throws IOException {\n    getStreamer().waitAndQueuePacket(currentPacket);\n    currentPacket = null;\n  }","sourceCodeStart":471,"sourceCodeEnd":507,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L471-L507","documentation":"writeChunkPrepare() also validates the checksum bytes accompanying each chunk: cklen must be 0 or exactly getChecksumSize() (e.g. 4 bytes for CRC32/CRC32C), otherwise IOException('writeChunk() checksum size is supposed to be X but found to be Y'). This guards the invariant that the caller's checksummer and the stream's checksum type agree - a mismatch would silently corrupt block checksums, so the stream refuses.","triggerScenarios":"Appending with a different checksum type than the file was created with (e.g. file written with CRC32C, appender configured/constructed with CRC32 whose encoder emits a different length, or NULL-checksum files appended with real checksums); direct writeChunk calls passing hand-built checksum arrays of the wrong width; wrapper stream computing checksums with a differently-sized DataChecksum.","commonSituations":"Client config or code drift between the tool that created the file and the tool that appends (checksum type/size changed after an upgrade); homegrown ChecksumFileSystem-style layers using their own DataChecksum; files written with per-file NULL/RAW checksums being appended by stock writers.","solutions":["On append, pass null as the ChecksumOpt (and don't set a conflicting client dfs bytes-per-checksum/checksum type) so the stream adopts the file's existing checksum layout.","If you build DataChecksum yourself for a wrapper, construct it with the same type as the underlying file/stream rather than a locally configured one.","Audit any code calling writeChunk() directly: checksum arrays must be exactly getChecksumSize() bytes per chunk (0 only if checksums are genuinely absent).","If the file's checksum type is unknown, inspect it via the file's metadata (hdfs fsck / HdfsDataInputStream readChecksum) before choosing the appender's checksum."],"exampleFix":"// before\n// appender forces its own checksum layout\nChecksumOpt bad = new ChecksumOpt(DataChecksum.Type.NULL, 512);\nFSDataOutputStream out = fs.append(path); // wrapper then writes 4-byte CRCs -> mismatch\n\n// after\n// inherit the file's checksum layout on append\nFSDataOutputStream out = fs.append(path); // and drop the custom ChecksumOpt/DataChecksum wrapper\nout.write(data, 0, data.length);","handlingStrategy":"validation","validationCode":"int expectedChecksumSize = checksum.getChecksumSize(); // e.g. 4 for CRC32/CRC32C\nif (suppliedChecksumLen != 0 && suppliedChecksumLen != expectedChecksumSize) {\n  throw new IllegalStateException(\"checksum width \" + suppliedChecksumLen\n      + \" != stream checksum size \" + expectedChecksumSize);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["On append, inherit the file's checksum layout (null ChecksumOpt) rather than overriding it.","Do not switch checksum type (CRC32/CRC32C/NULL) between the create and the append toolchains.","Construct wrapper DataChecksum with the same type as the underlying file.","Keep client dfs.bytes-per-checksum identical across all tools touching the same files."],"tags":["hdfs","hdfs-client","write","checksum","chunk"],"backgroundTag":"checksum-size-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}