apache/hadoop · error · IOException

DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC

Error message

DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC

What it means

When combining block checksums, if a block after the first reports a different CRC algorithm (e.g., CRC32 vs CRC32C) the MD5MD5CRC path marks the file's crcType as MIXED, but COMPOSITE_CRC cannot represent mixed types and throws instead. So the tolerance is inverted relative to bytesPerCRC: COMPOSITE_CRC requires one uniform CRC algorithm across all blocks. Files end up with mixed CRC types via appends after dfs.checksum.type changed, or concatenation of files written with different algorithms.

Solutions

  1. Rewrite the file to a uniform checksum type (distcp without -p checksum, or hdfs dfs -get/-put) — the durable fix
  2. Fall back to dfs.checksum.combine.mode=MD5MD5CRC, which represents mixed files as DataChecksum.Type.MIXED instead of failing
  3. Keep dfs.checksum.type unchanged for the lifetime of files that may be appended to
  4. For distcp, verify both ends use the same dfs.checksum.type or skip checksum verification

Example fix

<!-- before: COMPOSITE_CRC cannot handle mixed CRC32/CRC32C blocks -->
<property>
  <name>dfs.checksum.combine.mode</name>
  <value>COMPOSITE_CRC</value>
</property>

<!-- after: MD5MD5CRC degrades to Type.MIXED rather than failing -->
<property>
  <name>dfs.checksum.combine.mode</name>
  <value>MD5MD5CRC</value>
</property>
Defensive patterns

Strategy: fallback

Validate before calling

// No public API exposes per-block CRC type cheaply; guard by policy:
// if a file predates a dfs.checksum.type change, verify with MD5MD5CRC
// (reports Type.MIXED) instead of COMPOSITE_CRC.
conf.set("dfs.checksum.combine.mode", "MD5MD5CRC");

Try / catch

try {
  conf.set("dfs.checksum.combine.mode", "COMPOSITE_CRC");
  return fs.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage().contains("MIXED is not supported for COMPOSITE_CRC")) {
    conf.set("dfs.checksum.combine.mode", "MD5MD5CRC");
    return fs.getFileChecksum(path);
  }
  throw e;
}

Prevention

When it happens

Trigger: getFileChecksum() with dfs.checksum.combine.mode=COMPOSITE_CRC on a file whose blocks mix CRC32 and CRC32C (or any two DataChecksum.Type values); distcp -pb with COMPOSITE_CRC between clusters configured with different dfs.checksum.type.

Common situations: dfs.checksum.type switched from CRC32 to CRC32C and old files later appended; hdfs dfs -concat of files written under different checksum algorithms; migration projects verifying checksums of historical data with the newer composite mode.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/11fd7c51200c77c5. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java:411

        setCrcPerBlock(cpb);
      }

      // read crc-type
      final DataChecksum.Type ct;
      if (checksumData.hasCrcType()) {
        ct = PBHelperClient.convert(checksumData.getCrcType());
      } else {
        LOG.debug("Retrieving checksum from an earlier-version DataNode: " +
            "inferring checksum by reading first byte");
        ct = getClient().inferChecksumTypeByReading(locatedBlock, datanode);
      }

      if (blockIdx == 0) {
        setCrcType(ct);
      } else if (getCrcType() != DataChecksum.Type.MIXED &&
          getCrcType() != ct) {
        if (getBlockChecksumType() == BlockChecksumType.COMPOSITE_CRC) {
          throw new IOException(
              "DataChecksum.Type.MIXED is not supported for COMPOSITE_CRC");
        } else {
          // if crc types are mixed in a file
          setCrcType(DataChecksum.Type.MIXED);
        }
      }

      if (blockIdx == 0) {
        LOG.debug("set bytesPerCRC={}, crcPerBlock={}",
            getBytesPerCRC(), getCrcPerBlock());
      }
    }

    /**
     * Parses out the raw blockChecksum bytes from {@code checksumData}
     * according to the blockChecksumType and populates the cumulative
     * blockChecksumBuf with it.
     *

View on GitHub (pinned to 2add963021)