apache/hadoop · error · IOException

DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC

Error message

DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC

What it means

When combining block checksums, if a block after the first reports a different CRC algorithm (e.g., CRC32 vs CRC32C) the MD5MD5CRC path marks the file's crcType as MIXED, but COMPOSITE_CRC cannot represent mixed types and throws instead. So the tolerance is inverted relative to bytesPerCRC: COMPOSITE_CRC requires one uniform CRC algorithm across all blocks. Files end up with mixed CRC types via appends after dfs.checksum.type changed, or concatenation of files written with different algorithms.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java:411

        setCrcPerBlock(cpb);
      }

      // read crc-type
      final DataChecksum.Type ct;
      if (checksumData.hasCrcType()) {
        ct = PBHelperClient.convert(checksumData.getCrcType());
      } else {
        LOG.debug("Retrieving checksum from an earlier-version DataNode: " +
            "inferring checksum by reading first byte");
        ct = getClient().inferChecksumTypeByReading(locatedBlock, datanode);
      }

      if (blockIdx == 0) {
        setCrcType(ct);
      } else if (getCrcType() != DataChecksum.Type.MIXED &&
          getCrcType() != ct) {
        if (getBlockChecksumType() == BlockChecksumType.COMPOSITE_CRC) {
          throw new IOException(
              "DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC");
        } else {
          // if crc types are mixed in a file
          setCrcType(DataChecksum.Type.MIXED);
        }
      }

      if (blockIdx == 0) {
        LOG.debug("set bytesPerCRC={}, crcPerBlock={}",
            getBytesPerCRC(), getCrcPerBlock());
      }
    }

    /**
     * Parses out the raw blockChecksum bytes from {@code checksumData}
     * according to the blockChecksumType and populates the cumulative
     * blockChecksumBuf with it.
     *

View on GitHub (pinned to 2add963021)

Solutions

1. Rewrite the file to a uniform checksum type (distcp without -p checksum, or hdfs dfs -get/-put) — the durable fix
2. Fall back to dfs.checksum.combine.mode=MD5MD5CRC, which represents mixed files as DataChecksum.Type.MIXED instead of failing
3. Keep dfs.checksum.type unchanged for the lifetime of files that may be appended to
4. For distcp, verify both ends use the same dfs.checksum.type or skip checksum verification

Example fix

<!-- before: COMPOSITE_CRC cannot handle mixed CRC32/CRC32C blocks -->
<property>
  <name>dfs.checksum.combine.mode</name>
  <value>COMPOSITE_CRC</value>
</property>

<!-- after: MD5MD5CRC degrades to Type.MIXED rather than failing -->
<property>
  <name>dfs.checksum.combine.mode</name>
  <value>MD5MD5CRC</value>
</property>

// No public API exposes per-block CRC type cheaply; guard by policy:
// if a file predates a dfs.checksum.type change, verify with MD5MD5CRC
// (reports Type.MIXED) instead of COMPOSITE_CRC.
conf.set("dfs.checksum.combine.mode", "MD5MD5CRC");

try {
  conf.set("dfs.checksum.combine.mode", "COMPOSITE_CRC");
  return fs.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage().contains("MIXED is not supported for COMPOSITE_CRC")) {
    conf.set("dfs.checksum.combine.mode", "MD5MD5CRC");
    return fs.getFileChecksum(path);
  }
  throw e;
}

When it happens

Trigger: getFileChecksum() with dfs.checksum.combine.mode=COMPOSITE_CRC on a file whose blocks mix CRC32 and CRC32C (or any two DataChecksum.Type values); distcp -pb with COMPOSITE_CRC between clusters configured with different dfs.checksum.type.

Common situations: dfs.checksum.type switched from CRC32 to CRC32C and old files later appended; hdfs dfs -concat of files written under different checksum algorithms; migration projects verifying checksums of historical data with the newer composite mode.

Related errors

• Invalid checksum type in dfs.checksum.type: {}
• Invalid checksum type: userOpt={}, default={}, effective=nul
• Invalid values: dfs.bytes-per-checksum (={}) must divide cel
• Unknown ChecksumCombineMode: {}
• Byte-per-checksum not matched: bpc={} but bytesPerCRC={}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/11fd7c51200c77c5. Report an issue: GitHub.