{"record":{"id":"da0d802b074704b5","repo":"apache/hadoop","slug":"unexpected-blockchecksumtype-s-expecting-compo-da0d80","errorCode":null,"errorMessage":"Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC","messagePattern":"Unexpected blockChecksumType '(.+?)', expecting COMPOSITE_CRC","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java","lineNumber":450,"sourceCode":"     */\n    String populateBlockChecksumBuf(OpBlockChecksumResponseProto checksumData)\n        throws IOException {\n      String blockChecksumForDebug = null;\n      switch (getBlockChecksumType()) {\n      case MD5CRC:\n        //read md5\n        final MD5Hash md5 = new MD5Hash(\n            checksumData.getBlockChecksum().toByteArray());\n        md5.write(getBlockChecksumBuf());\n        if (LOG.isDebugEnabled()) {\n          blockChecksumForDebug = md5.toString();\n        }\n        break;\n      case COMPOSITE_CRC:\n        BlockChecksumType returnedType = PBHelperClient.convert(\n            checksumData.getBlockChecksumOptions().getBlockChecksumType());\n        if (returnedType != BlockChecksumType.COMPOSITE_CRC) {\n          throw new IOException(String.format(\n              \"Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC\",\n              returnedType));\n        }\n        byte[] crcBytes = checksumData.getBlockChecksum().toByteArray();\n        if (LOG.isDebugEnabled()) {\n          blockChecksumForDebug = CrcUtil.toSingleCrcString(crcBytes);\n        }\n        getBlockChecksumBuf().write(crcBytes);\n        break;\n      default:\n        throw new IOException(\n            \"Unknown BlockChecksumType: \" + getBlockChecksumType());\n      }\n      return blockChecksumForDebug;\n    }\n  }\n\n  /**","sourceCodeStart":432,"sourceCodeEnd":468,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java#L432-L468","documentation":"When the client requests per-block checksums with BlockChecksumType.COMPOSITE_CRC, it verifies the DataNode honored the requested blockChecksumOptions; if the reply carries a different type (typically MD5CRC from an older DataNode that ignores the option), it throws. This is a client/DataNode protocol-capability mismatch: COMPOSITE_CRC block checksums were added in HDFS 2.9/3.0 (HDFS-13063/HDFS-14484 era), and pre-support DataNodes answer with the legacy MD5 block checksum.","triggerScenarios":"Client with dfs.checksum.combine.mode=COMPOSITE_CRC (or distcp requesting composite checksums) reading a block from a DataNode running a version without blockChecksumOptions support; rolling upgrades where some DataNodes are still on 2.x; mixed-version clusters mid-upgrade.","commonSituations":"Rolling upgrade from Hadoop 2.x to 3.x while distcp or applications use COMPOSITE_CRC; edge clients on new versions against an older cluster; third-party distributions that lag the blockChecksumOptions feature.","solutions":["During rolling upgrades use dfs.checksum.combine.mode=MD5MD5CRC until every DataNode is upgraded past 2.9/3.0","Complete the DataNode upgrade to a version that supports COMPOSITE_CRC block checksums","For distcp in mixed clusters, disable checksum verification or use the legacy combine mode","Check DataNode logs/version (hdfs dfsadmin -report) to find nodes answering with legacy MD5 block checksums"],"exampleFix":"# before: mixed-version cluster, composite requested\ndfs.checksum.combine.mode=COMPOSITE_CRC\n\n# after: safe during rolling upgrade\ndfs.checksum.combine.mode=MD5MD5CRC","handlingStrategy":"fallback","validationCode":"// Probe cluster DataNode versions before using composite checksums:\n// hdfs dfsadmin -report (or FSNamesystem stats) -> ensure all DNs >= 2.9/3.0\n// Then choose mode accordingly:\nboolean allDnsModern = checkDnVersionsAtLeast(\"3.0\");\nconf.set(\"dfs.checksum.combine.mode\", allDnsModern ? \"COMPOSITE_CRC\" : \"MD5MD5CRC\");","typeGuard":null,"tryCatchPattern":"try {\n  return fs.getFileChecksum(path);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"expecting COMPOSITE_CRC\")) {\n    // DataNode predates blockChecksumOptions; fall back during rolling upgrade\n    conf.set(\"dfs.checksum.combine.mode\", \"MD5MD5CRC\");\n    return fs.getFileChecksum(path);\n  }\n  throw e;\n}","preventionTips":["During rolling upgrades, configure MD5MD5CRC until every DataNode supports COMPOSITE_CRC","Automate version checks (dfsadmin -report) in pipelines that use composite checksums","For distcp across mixed clusters, skip or relax checksum verification"],"tags":["hdfs","checksum","rolling-upgrade","version-mismatch","datanode"],"backgroundTag":"client-server-version-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}