apache/hadoop · error · IOException

Unexpected packet data length for {} from {}: a partial chun

Error message

Unexpected packet data length for {} from {}: a partial chunk must be sent in an individual packet (data length = {} > bytesPerChecksum = {})

What it means

Error "Unexpected packet data length for {} from {}: a partial chunk must be sent in an individual packet (data length = {} > bytesPerChecksum = {})" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java:726

          boolean alignedOnDisk = partialChunkSizeOnDisk == 0;
          boolean alignedInPacket = firstByteInBlock % bytesPerChecksum == 0;

          // If the end of the on-disk data is not chunk-aligned, the last
          // checksum needs to be overwritten.
          boolean overwriteLastCrc = !alignedOnDisk && !shouldNotWriteChecksum;
          // If the starting offset of the packat data is at the last chunk
          // boundary of the data on disk, the partial checksum recalculation
          // can be skipped and the checksum supplied by the client can be used
          // instead. This reduces disk reads and cpu load.
          boolean doCrcRecalc = overwriteLastCrc &&
              (lastChunkBoundary != firstByteInBlock);

          // If this is a partial chunk, then verify that this is the only
          // chunk in the packet. If the starting offset is not chunk
          // aligned, the packet should terminate at or before the next
          // chunk boundary.
          if (!alignedInPacket && len > bytesPerChecksum) {
            throw new IOException("Unexpected packet data length for "
                +  block + " from " + inAddr + ": a partial chunk must be "
                + " sent in an individual packet (data length = " + len
                +  " > bytesPerChecksum = " + bytesPerChecksum + ")");
          }

          // If the last portion of the block file is not a full chunk,
          // then read in pre-existing partial data chunk and recalculate
          // the checksum so that the checksum calculation can continue
          // from the right state. If the client provided the checksum for
          // the whole chunk, this is not necessary.
          Checksum partialCrc = null;
          if (doCrcRecalc) {
            if (LOG.isDebugEnabled()) {
              LOG.debug("receivePacket for " + block 
                  + ": previous write did not end at the chunk boundary."
                  + " onDiskLen=" + onDiskLen);
            }
            long offsetInChecksum = BlockMetadataHeader.getHeaderSize() +

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure the client sends a partial final chunk as its own packet rather than coalescing it with other data; upgrade clients with this bug.
2. Verify client/DataNode version compatibility; this is a protocol-level violation by the sender.

When it happens

Trigger: A packet's data length exceeds one bytesPerChecksum unit while containing a partial chunk, violating the rule that partial chunks are sent in individual packets.

Common situations: A packet carried a partial chunk larger than bytesPerChecksum, which the protocol forbids. Usually caused by mismatched client/server chunk size settings.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fe51f81ab16badac. Report an issue: GitHub.