apache/hadoop · error · IOException

writeChunk() checksum size is supposed to be {} but found to

Error message

writeChunk() checksum size is supposed to be {} but found to be {}

What it means

writeChunkPrepare() also validates the checksum bytes accompanying each chunk: cklen must be 0 or exactly getChecksumSize() (e.g. 4 bytes for CRC32/CRC32C), otherwise IOException('writeChunk() checksum size is supposed to be X but found to be Y'). This guards the invariant that the caller's checksummer and the stream's checksum type agree - a mismatch would silently corrupt block checksums, so the stream refuses.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:489

    // If packet is full, enqueue it for transmission
    if (currentPacket.getNumChunks() == currentPacket.getMaxChunks() ||
            getStreamer().getBytesCurBlock() == blockSize) {
      enqueueCurrentPacketFull();
    }
  }

  private synchronized void writeChunkPrepare(int buflen,
      int ckoff, int cklen) throws IOException {
    dfsClient.checkOpen();
    checkClosed();

    if (buflen > bytesPerChecksum) {
      throw new IOException("writeChunk() buffer size is " + buflen +
                            " is larger than supported  bytesPerChecksum " +
                            bytesPerChecksum);
    }
    if (cklen != 0 && cklen != getChecksumSize()) {
      throw new IOException("writeChunk() checksum size is supposed to be " +
                            getChecksumSize() + " but found to be " + cklen);
    }

    if (currentPacket == null) {
      currentPacket = createPacket(packetSize, chunksPerPacket, getStreamer()
          .getBytesCurBlock(), getStreamer().getAndIncCurrentSeqno(), false);
      DFSClient.LOG.debug("WriteChunk allocating new packet seqno={},"
              + " src={}, packetSize={}, chunksPerPacket={}, bytesCurBlock={},"
              + " output stream={}",
          currentPacket.getSeqno(), src, packetSize, chunksPerPacket,
          getStreamer().getBytesCurBlock(), this);
    }
  }

  void enqueueCurrentPacket() throws IOException {
    getStreamer().waitAndQueuePacket(currentPacket);
    currentPacket = null;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. On append, pass null as the ChecksumOpt (and don't set a conflicting client dfs bytes-per-checksum/checksum type) so the stream adopts the file's existing checksum layout.
2. If you build DataChecksum yourself for a wrapper, construct it with the same type as the underlying file/stream rather than a locally configured one.
3. Audit any code calling writeChunk() directly: checksum arrays must be exactly getChecksumSize() bytes per chunk (0 only if checksums are genuinely absent).
4. If the file's checksum type is unknown, inspect it via the file's metadata (hdfs fsck / HdfsDataInputStream readChecksum) before choosing the appender's checksum.

Example fix

// before
// appender forces its own checksum layout
ChecksumOpt bad = new ChecksumOpt(DataChecksum.Type.NULL, 512);
FSDataOutputStream out = fs.append(path); // wrapper then writes 4-byte CRCs -> mismatch

// after
// inherit the file's checksum layout on append
FSDataOutputStream out = fs.append(path); // and drop the custom ChecksumOpt/DataChecksum wrapper
out.write(data, 0, data.length);

int expectedChecksumSize = checksum.getChecksumSize(); // e.g. 4 for CRC32/CRC32C
if (suppliedChecksumLen != 0 && suppliedChecksumLen != expectedChecksumSize) {
  throw new IllegalStateException("checksum width " + suppliedChecksumLen
      + " != stream checksum size " + expectedChecksumSize);
}

When it happens

Trigger: Appending with a different checksum type than the file was created with (e.g. file written with CRC32C, appender configured/constructed with CRC32 whose encoder emits a different length, or NULL-checksum files appended with real checksums); direct writeChunk calls passing hand-built checksum arrays of the wrong width; wrapper stream computing checksums with a differently-sized DataChecksum.

Common situations: Client config or code drift between the tool that created the file and the tool that appends (checksum type/size changed after an upgrade); homegrown ChecksumFileSystem-style layers using their own DataChecksum; files written with per-file NULL/RAW checksums being appended by stock writers.

Related errors

• writeChunk() buffer size is {} is larger than supported byt
• Invalid value: bytesPerChecksum = {} <= 0
• Invalid values: dfs.bytes-per-checksum (={}) must divide blo
• Invalid values: dfs.bytes-per-checksum (={}) must divide cel
• Could not create DataChecksum from the byte array of length

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e3ed75d6e9c6db1b. Report an issue: GitHub.