{"record":{"id":"11fd7c51200c77c5","repo":"apache/hadoop","slug":"datachecksum-type-mixed-is-not-supported-for-compo","errorCode":null,"errorMessage":"DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC","messagePattern":"DataChecksum\\.Type\\.MIXED is not supported for COMPOSITE_CRC","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java","lineNumber":411,"sourceCode":"        setCrcPerBlock(cpb);\n      }\n\n      // read crc-type\n      final DataChecksum.Type ct;\n      if (checksumData.hasCrcType()) {\n        ct = PBHelperClient.convert(checksumData.getCrcType());\n      } else {\n        LOG.debug(\"Retrieving checksum from an earlier-version DataNode: \" +\n            \"inferring checksum by reading first byte\");\n        ct = getClient().inferChecksumTypeByReading(locatedBlock, datanode);\n      }\n\n      if (blockIdx == 0) {\n        setCrcType(ct);\n      } else if (getCrcType() != DataChecksum.Type.MIXED &&\n          getCrcType() != ct) {\n        if (getBlockChecksumType() == BlockChecksumType.COMPOSITE_CRC) {\n          throw new IOException(\n              \"DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC\");\n        } else {\n          // if crc types are mixed in a file\n          setCrcType(DataChecksum.Type.MIXED);\n        }\n      }\n\n      if (blockIdx == 0) {\n        LOG.debug(\"set bytesPerCRC={}, crcPerBlock={}\",\n            getBytesPerCRC(), getCrcPerBlock());\n      }\n    }\n\n    /**\n     * Parses out the raw blockChecksum bytes from {@code checksumData}\n     * according to the blockChecksumType and populates the cumulative\n     * blockChecksumBuf with it.\n     *","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java#L393-L429","documentation":"When combining block checksums, if a block after the first reports a different CRC algorithm (e.g., CRC32 vs CRC32C) the MD5MD5CRC path marks the file's crcType as MIXED, but COMPOSITE_CRC cannot represent mixed types and throws instead. So the tolerance is inverted relative to bytesPerCRC: COMPOSITE_CRC requires one uniform CRC algorithm across all blocks. Files end up with mixed CRC types via appends after dfs.checksum.type changed, or concatenation of files written with different algorithms.","triggerScenarios":"getFileChecksum() with dfs.checksum.combine.mode=COMPOSITE_CRC on a file whose blocks mix CRC32 and CRC32C (or any two DataChecksum.Type values); distcp -pb with COMPOSITE_CRC between clusters configured with different dfs.checksum.type.","commonSituations":"dfs.checksum.type switched from CRC32 to CRC32C and old files later appended; hdfs dfs -concat of files written under different checksum algorithms; migration projects verifying checksums of historical data with the newer composite mode.","solutions":["Rewrite the file to a uniform checksum type (distcp without -p checksum, or hdfs dfs -get/-put) — the durable fix","Fall back to dfs.checksum.combine.mode=MD5MD5CRC, which represents mixed files as DataChecksum.Type.MIXED instead of failing","Keep dfs.checksum.type unchanged for the lifetime of files that may be appended to","For distcp, verify both ends use the same dfs.checksum.type or skip checksum verification"],"exampleFix":"<!-- before: COMPOSITE_CRC cannot handle mixed CRC32/CRC32C blocks -->\n<property>\n  <name>dfs.checksum.combine.mode</name>\n  <value>COMPOSITE_CRC</value>\n</property>\n\n<!-- after: MD5MD5CRC degrades to Type.MIXED rather than failing -->\n<property>\n  <name>dfs.checksum.combine.mode</name>\n  <value>MD5MD5CRC</value>\n</property>","handlingStrategy":"fallback","validationCode":"// No public API exposes per-block CRC type cheaply; guard by policy:\n// if a file predates a dfs.checksum.type change, verify with MD5MD5CRC\n// (reports Type.MIXED) instead of COMPOSITE_CRC.\nconf.set(\"dfs.checksum.combine.mode\", \"MD5MD5CRC\");","typeGuard":null,"tryCatchPattern":"try {\n  conf.set(\"dfs.checksum.combine.mode\", \"COMPOSITE_CRC\");\n  return fs.getFileChecksum(path);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"MIXED is not supported for COMPOSITE_CRC\")) {\n    conf.set(\"dfs.checksum.combine.mode\", \"MD5MD5CRC\");\n    return fs.getFileChecksum(path);\n  }\n  throw e;\n}","preventionTips":["Keep dfs.checksum.type stable for the lifetime of appendable files","When verifying checksums of files that may mix CRC32/CRC32C, prefer MD5MD5CRC combine mode","After concat operations, remember checksum heterogeneity is inherited — re-verify with a tolerant mode"],"tags":["hdfs","checksum","erasure-coding-unrelated","configuration","crc32c"],"backgroundTag":"checksum-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}