apache/hadoop · error · IOException

Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC

Error message

Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC

What it means

When the client requests per-block checksums with BlockChecksumType.COMPOSITE_CRC, it verifies the DataNode honored the requested blockChecksumOptions; if the reply carries a different type (typically MD5CRC from an older DataNode that ignores the option), it throws. This is a client/DataNode protocol-capability mismatch: COMPOSITE_CRC block checksums were added in HDFS 2.9/3.0 (HDFS-13063/HDFS-14484 era), and pre-support DataNodes answer with the legacy MD5 block checksum.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java:450

     */
    String populateBlockChecksumBuf(OpBlockChecksumResponseProto checksumData)
        throws IOException {
      String blockChecksumForDebug = null;
      switch (getBlockChecksumType()) {
      case MD5CRC:
        //read md5
        final MD5Hash md5 = new MD5Hash(
            checksumData.getBlockChecksum().toByteArray());
        md5.write(getBlockChecksumBuf());
        if (LOG.isDebugEnabled()) {
          blockChecksumForDebug = md5.toString();
        }
        break;
      case COMPOSITE_CRC:
        BlockChecksumType returnedType = PBHelperClient.convert(
            checksumData.getBlockChecksumOptions().getBlockChecksumType());
        if (returnedType != BlockChecksumType.COMPOSITE_CRC) {
          throw new IOException(String.format(
              "Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC",
              returnedType));
        }
        byte[] crcBytes = checksumData.getBlockChecksum().toByteArray();
        if (LOG.isDebugEnabled()) {
          blockChecksumForDebug = CrcUtil.toSingleCrcString(crcBytes);
        }
        getBlockChecksumBuf().write(crcBytes);
        break;
      default:
        throw new IOException(
            "Unknown BlockChecksumType: " + getBlockChecksumType());
      }
      return blockChecksumForDebug;
    }
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. During rolling upgrades use dfs.checksum.combine.mode=MD5MD5CRC until every DataNode is upgraded past 2.9/3.0
2. Complete the DataNode upgrade to a version that supports COMPOSITE_CRC block checksums
3. For distcp in mixed clusters, disable checksum verification or use the legacy combine mode
4. Check DataNode logs/version (hdfs dfsadmin -report) to find nodes answering with legacy MD5 block checksums

Example fix

# before: mixed-version cluster, composite requested
dfs.checksum.combine.mode=COMPOSITE_CRC

# after: safe during rolling upgrade
dfs.checksum.combine.mode=MD5MD5CRC

// Probe cluster DataNode versions before using composite checksums:
// hdfs dfsadmin -report (or FSNamesystem stats) -> ensure all DNs >= 2.9/3.0
// Then choose mode accordingly:
boolean allDnsModern = checkDnVersionsAtLeast("3.0");
conf.set("dfs.checksum.combine.mode", allDnsModern ? "COMPOSITE_CRC" : "MD5MD5CRC");

try {
  return fs.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage().contains("expecting COMPOSITE_CRC")) {
    // DataNode predates blockChecksumOptions; fall back during rolling upgrade
    conf.set("dfs.checksum.combine.mode", "MD5MD5CRC");
    return fs.getFileChecksum(path);
  }
  throw e;
}

When it happens

Trigger: Client with dfs.checksum.combine.mode=COMPOSITE_CRC (or distcp requesting composite checksums) reading a block from a DataNode running a version without blockChecksumOptions support; rolling upgrades where some DataNodes are still on 2.x; mixed-version clusters mid-upgrade.

Common situations: Rolling upgrade from Hadoop 2.x to 3.x while distcp or applications use COMPOSITE_CRC; edge clients on new versions against an older cluster; third-party distributions that lag the blockChecksumOptions feature.

Related errors

• Version Mismatch (Expected: {}, Received: {} )
• Meta-data not found for {block}
• Checksum failed at {failedPos} for replica: {replica}
• BUG: The stored LV = {} is newer than the supported LV = {}
• Meta file for {} not found.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/da0d802b074704b5. Report an issue: GitHub.