apache/hadoop · error · IOException

Byte-per-checksum not matched: bpc={} but bytesPerCRC={}

Error message

Byte-per-checksum not matched: bpc={} but bytesPerCRC={}

What it means

While computing an MD5MD5CRC file checksum, the client reads each block's checksum header and requires bytesPerCRC (io.bytes.per.checksum) to be identical across all blocks — the MD5-of-MD5s combination is only defined for uniform chunking. If a later block reports a different bpc than the first block and the mode is not COMPOSITE_CRC, it throws this IOException. COMPOSITE_CRC is specifically designed to tolerate varying bytesPerCRC, which is why the code just logs a warning and continues in that mode.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java:385

    void extractChecksumProperties(
        OpBlockChecksumResponseProto checksumData,
        LocatedBlock locatedBlock,
        DatanodeInfo datanode,
        int blockIdx)
        throws IOException {
      //read byte-per-checksum
      final int bpc = checksumData.getBytesPerCrc();
      if (blockIdx == 0) { //first block
        setBytesPerCRC(bpc);
      } else if (bpc != getBytesPerCRC()) {
        if (getBlockChecksumType() == BlockChecksumType.COMPOSITE_CRC) {
          LOG.warn(
              "Current bytesPerCRC={} doesn't match next bpc={}, but "
              + "continuing anyway because we're using COMPOSITE_CRC. "
              + "If trying to preserve CHECKSUMTYPE, only the current "
              + "bytesPerCRC will be preserved.", getBytesPerCRC(), bpc);
        } else {
          throw new IOException("Byte-per-checksum not matched: bpc=" + bpc
              + " but bytesPerCRC=" + getBytesPerCRC());
        }
      }

      //read crc-per-block
      final long cpb = checksumData.getCrcPerBlock();
      if (getLocatedBlocks().size() > 1 && blockIdx == 0) {
        setCrcPerBlock(cpb);
      }

      // read crc-type
      final DataChecksum.Type ct;
      if (checksumData.hasCrcType()) {
        ct = PBHelperClient.convert(checksumData.getCrcType());
      } else {
        LOG.debug("Retrieving checksum from an earlier-version DataNode: " +
            "inferring checksum by reading first byte");
        ct = getClient().inferChecksumTypeByReading(locatedBlock, datanode);

View on GitHub (pinned to 2add963021)

Solutions

  1. Set dfs.checksum.combine.mode=COMPOSITE_CRC on the client (and both sides for distcp) — it supports differing bytesPerCRC by design
  2. For distcp, drop checksum verification (-skip-checksum-difference) or use COMPOSITE_CRC on both clusters
  3. Rewrite the file (hdfs dfs -get/-put, or distcp without -p checksum) to normalize all blocks to current checksum settings
  4. Prevent recurrence: keep io.bytes.per.checksum stable for the lifetime of files that will be appended to

Example fix

<!-- before -->
<property>
  <name>dfs.checksum.combine.mode</name>
  <value>MD5MD5CRC</value>
</property>

<!-- after: tolerate per-block bytesPerCRC differences -->
<property>
  <name>dfs.checksum.combine.mode</name>
  <value>COMPOSITE_CRC</value>
</property>
Defensive patterns

Strategy: fallback

Validate before calling

// Before a checksum job over legacy data, check per-block checksum settings
// via a probe read: compute checksum of a small range and on failure switch mode.
// Simplest up-front guard: choose COMPOSITE_CRC when files may be heterogeneous.
conf.set("dfs.checksum.combine.mode", "COMPOSITE_CRC");

Try / catch

try {
  return fs.getFileChecksum(path); // MD5MD5CRC default
} catch (IOException e) {
  if (e.getMessage().contains("Byte-per-checksum not matched")) {
    conf.set("dfs.checksum.combine.mode", "COMPOSITE_CRC");
    return fs.getFileChecksum(path); // tolerant combine mode
  }
  throw e;
}

Prevention

When it happens

Trigger: getFileChecksum()/DistCp -crc over a file whose blocks were written with different io.bytes.per.checksum values — e.g., appended after the cluster/client checksum config changed, files created pre- and post-config-change, or concatenated files (hdfs dfs -concat) with heterogeneous chunk sizes; the first block fixes bytesPerCRC and a subsequent block disagrees.

Common situations: Appends to old files after io.bytes.per.checksum was changed; distcp between clusters with different checksum settings using -crc/-p checksum; verifying checksums of archived data written under legacy 512-byte settings vs newer custom values.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5660873d35422808. Report an issue: GitHub.