{"record":{"id":"14728432cdfc27b0","repo":"apache/hadoop","slug":"writechunk-buffer-size-is-is-larger-than-supp","errorCode":null,"errorMessage":"writeChunk() buffer size is {} is larger than supported  bytesPerChecksum {}","messagePattern":"writeChunk\\(\\) buffer size is (.+?) is larger than supported  bytesPerChecksum (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":484,"sourceCode":"    currentPacket.writeChecksum(checksum, ckoff, cklen);\n    currentPacket.writeData(buffer, len);\n    currentPacket.incNumChunks();\n    getStreamer().incBytesCurBlock(len);\n\n    // If packet is full, enqueue it for transmission\n    if (currentPacket.getNumChunks() == currentPacket.getMaxChunks() ||\n            getStreamer().getBytesCurBlock() == blockSize) {\n      enqueueCurrentPacketFull();\n    }\n  }\n\n  private synchronized void writeChunkPrepare(int buflen,\n      int ckoff, int cklen) throws IOException {\n    dfsClient.checkOpen();\n    checkClosed();\n\n    if (buflen > bytesPerChecksum) {\n      throw new IOException(\"writeChunk() buffer size is \" + buflen +\n                            \" is larger than supported  bytesPerChecksum \" +\n                            bytesPerChecksum);\n    }\n    if (cklen != 0 && cklen != getChecksumSize()) {\n      throw new IOException(\"writeChunk() checksum size is supposed to be \" +\n                            getChecksumSize() + \" but found to be \" + cklen);\n    }\n\n    if (currentPacket == null) {\n      currentPacket = createPacket(packetSize, chunksPerPacket, getStreamer()\n          .getBytesCurBlock(), getStreamer().getAndIncCurrentSeqno(), false);\n      DFSClient.LOG.debug(\"WriteChunk allocating new packet seqno={},\"\n              + \" src={}, packetSize={}, chunksPerPacket={}, bytesCurBlock={},\"\n              + \" output stream={}\",\n          currentPacket.getSeqno(), src, packetSize, chunksPerPacket,\n          getStreamer().getBytesCurBlock(), this);\n    }\n  }","sourceCodeStart":466,"sourceCodeEnd":502,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L466-L502","documentation":"writeChunkPrepare() validates every chunk submitted through the chunk-based write path (used by FSOutputSummer's checksumming write loop): the data buffer length must not exceed bytesPerChecksum, otherwise IOException('writeChunk() buffer size is X is larger than supported bytesPerChecksum Y'). Under normal use the checksummer slices writes into chunks of exactly bytesPerChecksum, so this fires only when something bypasses or disagrees with that slicing - direct writeChunk calls or a wrapping layer with a different chunk size.","triggerScenarios":"Code invoking writeChunk/b, off, len, checksum) directly with len > bytesPerChecksum; a ChecksumFileSystem-style wrapper (LocalFileSystem, ChecksumFileSystem subclasses) built with a different bytesPerChecksum than the one the DFS stream was opened with; append where the caller's checksum layout (e.g. 256) is finer than the file's (512) and chunks cross the boundary.","commonSituations":"Legacy MapReduce/record-writer code ported from old HDFS APIs that wrote chunks manually; custom FileSystem wrappers layered over DistributedFileSystem with their own DataChecksum; a client config change of dfs.bytes-per-checksum between the file's creation and later appends by chunk-based writers.","solutions":["Do not call writeChunk() directly - write through write(byte[],off,len) / hflush and let the checksummer slice correctly.","If you wrap the stream, build your DataChecksum with exactly the same bytesPerChecksum as the underlying DFS stream (inherit: pass null ChecksumOpt on create/append so server/file layout wins).","On append, re-check that the client's dfs.bytes-per-checksum matches the file; when unsure, unset it client-side and let the NameNode return the file's checksum layout."],"exampleFix":"// before\n// chunk-writer with stale layout\nout.writeChunk(data, 0, 1024, checksum4Bytes); // bytesPerChecksum is 512 -> throws\n\n// after\nout.write(data, 0, 1024); // FSOutputSummer slices into 512-byte chunks itself","handlingStrategy":"validation","validationCode":"int bytesPerChecksum = checksumOpt != null ? checksumOpt.getBytesPerChecksum() : 512;\n// any chunked write you perform yourself must respect the stream's chunk size\nif (chunkLen > bytesPerChecksum) {\n  throw new IllegalStateException(\"chunk \" + chunkLen + \" > bytesPerChecksum \" + bytesPerChecksum);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never call writeChunk() directly - use write(byte[],int,int) and let FSOutputSummer slice chunks.","Build wrapper streams with the same DataChecksum (type and size) as the underlying DFS stream.","Pass null ChecksumOpt on create/append to inherit the server/file layout instead of forcing one.","Re-check dfs.bytes-per-checksum consistency between creating and appending tools."],"tags":["hdfs","hdfs-client","write","checksum","chunk"],"backgroundTag":"checksum-chunk-size-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}