apache/hadoop · error · IOException

writeChunk() buffer size is {} is larger than supported byt

Error message

writeChunk() buffer size is {} is larger than supported  bytesPerChecksum {}

What it means

writeChunkPrepare() validates every chunk submitted through the chunk-based write path (used by FSOutputSummer's checksumming write loop): the data buffer length must not exceed bytesPerChecksum, otherwise IOException('writeChunk() buffer size is X is larger than supported bytesPerChecksum Y'). Under normal use the checksummer slices writes into chunks of exactly bytesPerChecksum, so this fires only when something bypasses or disagrees with that slicing - direct writeChunk calls or a wrapping layer with a different chunk size.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:484

    currentPacket.writeChecksum(checksum, ckoff, cklen);
    currentPacket.writeData(buffer, len);
    currentPacket.incNumChunks();
    getStreamer().incBytesCurBlock(len);

    // If packet is full, enqueue it for transmission
    if (currentPacket.getNumChunks() == currentPacket.getMaxChunks() ||
            getStreamer().getBytesCurBlock() == blockSize) {
      enqueueCurrentPacketFull();
    }
  }

  private synchronized void writeChunkPrepare(int buflen,
      int ckoff, int cklen) throws IOException {
    dfsClient.checkOpen();
    checkClosed();

    if (buflen > bytesPerChecksum) {
      throw new IOException("writeChunk() buffer size is " + buflen +
                            " is larger than supported  bytesPerChecksum " +
                            bytesPerChecksum);
    }
    if (cklen != 0 && cklen != getChecksumSize()) {
      throw new IOException("writeChunk() checksum size is supposed to be " +
                            getChecksumSize() + " but found to be " + cklen);
    }

    if (currentPacket == null) {
      currentPacket = createPacket(packetSize, chunksPerPacket, getStreamer()
          .getBytesCurBlock(), getStreamer().getAndIncCurrentSeqno(), false);
      DFSClient.LOG.debug("WriteChunk allocating new packet seqno={},"
              + " src={}, packetSize={}, chunksPerPacket={}, bytesCurBlock={},"
              + " output stream={}",
          currentPacket.getSeqno(), src, packetSize, chunksPerPacket,
          getStreamer().getBytesCurBlock(), this);
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Do not call writeChunk() directly - write through write(byte[],off,len) / hflush and let the checksummer slice correctly.
2. If you wrap the stream, build your DataChecksum with exactly the same bytesPerChecksum as the underlying DFS stream (inherit: pass null ChecksumOpt on create/append so server/file layout wins).
3. On append, re-check that the client's dfs.bytes-per-checksum matches the file; when unsure, unset it client-side and let the NameNode return the file's checksum layout.

Example fix

// before
// chunk-writer with stale layout
out.writeChunk(data, 0, 1024, checksum4Bytes); // bytesPerChecksum is 512 -> throws

// after
out.write(data, 0, 1024); // FSOutputSummer slices into 512-byte chunks itself

int bytesPerChecksum = checksumOpt != null ? checksumOpt.getBytesPerChecksum() : 512;
// any chunked write you perform yourself must respect the stream's chunk size
if (chunkLen > bytesPerChecksum) {
  throw new IllegalStateException("chunk " + chunkLen + " > bytesPerChecksum " + bytesPerChecksum);
}

When it happens

Trigger: Code invoking writeChunk/b, off, len, checksum) directly with len > bytesPerChecksum; a ChecksumFileSystem-style wrapper (LocalFileSystem, ChecksumFileSystem subclasses) built with a different bytesPerChecksum than the one the DFS stream was opened with; append where the caller's checksum layout (e.g. 256) is finer than the file's (512) and chunks cross the boundary.

Common situations: Legacy MapReduce/record-writer code ported from old HDFS APIs that wrote chunks manually; custom FileSystem wrappers layered over DistributedFileSystem with their own DataChecksum; a client config change of dfs.bytes-per-checksum between the file's creation and later appends by chunk-based writers.

Related errors

• writeChunk() checksum size is supposed to be {} but found to
• Invalid value: bytesPerChecksum = {} <= 0
• Invalid values: dfs.bytes-per-checksum (={}) must divide blo
• Invalid values: dfs.bytes-per-checksum (={}) must divide cel
• Could not create DataChecksum from the byte array of length

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/14728432cdfc27b0. Report an issue: GitHub.