apache/hadoop · error · IOException

Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC

Error message

Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC

What it means

During a COMPOSITE_CRC block-group checksum the DataNode asked a peer DN for a per-block composite CRC, but the reply's echoed BlockChecksumOptions reported a different type. The helper validates the echoed type because an older peer silently answers MD5CRC instead of honoring the requested COMPOSITE_CRC.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockChecksumHelper.java:667

        }

        setOrVerifyChecksumProperties(blockIdx, checksumData.getBytesPerCrc(),
            checksumData.getCrcPerBlock(), ct);

        switch (groupChecksumType) {
        case MD5CRC:
          //read md5
          final MD5Hash md5 =
              new MD5Hash(checksumData.getBlockChecksum().toByteArray());
          md5.write(blockChecksumBuf);
          LOG.debug("got reply from datanode:{}, md5={}",
              targetDatanode, md5);
          break;
        case COMPOSITE_CRC:
          BlockChecksumType returnedType = PBHelperClient.convert(
              checksumData.getBlockChecksumOptions().getBlockChecksumType());
          if (returnedType != BlockChecksumType.COMPOSITE_CRC) {
            throw new IOException(String.format(
                "Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC",
                returnedType));
          }
          byte[] checksumBytes =
              checksumData.getBlockChecksum().toByteArray();
          blockChecksumBuf.write(checksumBytes, 0, checksumBytes.length);
          if (LOG.isDebugEnabled()) {
            LOG.debug("got reply from datanode:{} for blockIdx:{}, checksum:{}",
                targetDatanode, blockIdx,
                CrcUtil.toMultiCrcString(checksumBytes));
          }
          break;
        default:
          throw new IOException(
              "Unknown BlockChecksumType: " + groupChecksumType);
        }
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Finish the rolling upgrade so every DataNode supports COMPOSITE_CRC before clients request it
2. Until then, run fileChecksum with MD5CRC (set dfs.client.files.checksum-type or the client option) — it works on mixed versions
3. If versions are already aligned, inspect the peer DN for protocol corruption or a stale jar

Example fix

// before
BlockChecksumOptions opts =
    new BlockChecksumOptions(BlockChecksumType.COMPOSITE_CRC, cellSize);

// after (until all DNs are upgraded)
BlockChecksumOptions opts =
    new BlockChecksumOptions(BlockChecksumType.MD5CRC, cellSize);

// Before requesting COMPOSITE_CRC on an EC file, confirm every replica-hosting
// DN runs a release that supports it (e.g. via DatanodeInfo version):
for (DatanodeInfo dn : locatedBlock.getLocations()) {
  if (!supportsCompositeCrc(dn.getSoftwareVersion())) {
    useComposite = false; break;
  }
}

try {
  digest = getFileChecksum(ecPath, COMPOSITE_CRC);
} catch (IOException e) {
  if (e.getMessage().contains("expecting COMPOSITE_CRC")) {
    digest = getFileChecksum(ecPath, MD5CRC); // mixed-version safe
  } else { throw e; }
}

When it happens

Trigger: An EC file's block group spans a DataNode running a release without COMPOSITE_CRC support, so it ignores the requested type and returns MD5CRC; also possible with a corrupted or hand-crafted BlockOpResponseProto.

Common situations: Rolling upgrade mid-way across DNs hosting one striped file; mixed-version cluster; distcp/verification tooling requesting composite CRCs too early.

Related errors

• Unrecognized BlockChecksumType: %s
• Unknown status: ${status}, message: ${message}
• {this}: invalid slot index {slotIdx}
• The value %d does not map to a valid checksum Type
• Version Mismatch (Expected: {}, Received: {} )

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cab9716f417b79e4. Report an issue: GitHub.