apache/hadoop · error · IOException

BlockChecksumType COMPOSITE_CRC doesn't support MIXED underl

Error message

BlockChecksumType COMPOSITE_CRC doesn't support MIXED underlying types; previous block was %s, next block is %s

What it means

COMPOSITE_CRC requires the underlying CRC algorithm (DataChecksum.Type, e.g. CRC32 vs CRC32C) to be identical across all internal blocks of the group; the Nth block reported a different type. Unlike MD5CRC, composite folding cannot represent a MIXED group, so it throws instead of silently producing a wrong digest.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockChecksumHelper.java:752

        setBytesPerCRC(bpc);
      } else if (bpc != getBytesPerCRC()) {
        throw new IOException("Byte-per-checksum not matched: bpc=" + bpc
            + " but bytesPerCRC=" + getBytesPerCRC());
      }

      //read crc-per-block
      if (blockIdx == 0) {
        setCrcPerBlock(cpb);
      }

      if (blockIdx == 0) { // first block
        setCrcType(ct);
      } else if (getCrcType() != DataChecksum.Type.MIXED &&
          getCrcType() != ct) {
        BlockChecksumType groupChecksumType =
            getBlockChecksumOptions().getBlockChecksumType();
        if (groupChecksumType == BlockChecksumType.COMPOSITE_CRC) {
          throw new IOException(String.format(
              "BlockChecksumType COMPOSITE_CRC doesn't support MIXED "
              + "underlying types; previous block was %s, next block is %s",
              getCrcType(), ct));
        } else {
          setCrcType(DataChecksum.Type.MIXED);
        }
      }

      if (blockIdx == 0) {
        LOG.debug("set bytesPerCRC={}, crcPerBlock={}", getBytesPerCRC(),
            getCrcPerBlock());
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Use MD5CRC for checksumming the affected file (MIXED is representable there)
2. Keep the checksum type constant cluster-wide and never change it for files that will be appended
3. Rewrite affected files (distcp) so all internal blocks share one type

// Prefer MD5CRC for files that could predate a checksum-type change:
BlockChecksumOptions opts = fileMayPredateConfigChange
    ? MD5CRC_OPTS : COMPOSITE_CRC_OPTS;

try {
  cs = fs.getFileChecksum(path, COMPOSITE_CRC_OPTS);
} catch (IOException e) {
  if (e.getMessage().contains("doesn't support MIXED")) {
    cs = fs.getFileChecksum(path, MD5CRC_OPTS);
  } else { throw e; }
}

When it happens

Trigger: The checksum type used at write time (per-file create option / dfs.checksum.type) differed between internal blocks of one striped file — e.g. type changed between original write and a later append, or blocks written by DNs with different configs.

Common situations: Changing dfs.checksum.type on an existing cluster; appending to old files after a config change; EC files written across a config migration.

Related errors

• Byte-per-checksum not matched: bpc={} but bytesPerCRC={}
• FileSystem ${item.fs.getUri()} does not support Erasure Codi
• Cannot create a secured connection if DataNode listens on un
• All negative block group IDs are used, growing into positive
• Unknown BlockChecksumType: " + groupChecksumType

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/49f6b1859a3e3568. Report an issue: GitHub.