apache/hadoop · error · IOException

Checksum mismatch between {} and {}.

Error message

Checksum mismatch between {} and {}.

What it means

After a copy with lengths matching, DistCpUtils compares source and target checksums; on mismatch it throws an IOException built around CHECKSUM_MISMATCH_ERROR_MSG naming both files, plus remediation hints: file-level checksum validation via -Ddfs.checksum.combine.mode=COMPOSITE_CRC when block sizes or filesystems differ, or -skipcrccheck to skip validation. Since lengths already matched, this is either genuinely different bytes or an incomparable per-block checksum comparison.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java:660

              .append("Their checksum algorithms may be incompatible");
          addSkipHint = true;
        } else if (sourceFS.getFileStatus(source).getBlockSize() !=
            targetFS.getFileStatus(target).getBlockSize()) {
          errorMessage.append(" Source and target differ in block-size.\n")
              .append(" Use -pb to preserve block-sizes during copy.");
          addSkipHint = true;
        }
        if (addSkipHint) {
          errorMessage
              .append(" You can choose file-level checksum validation via "
                  + "-Ddfs.checksum.combine.mode=COMPOSITE_CRC when block-sizes"
                  + " or filesystems are different.")
              .append(" Or you can skip checksum-checks altogether "
                  + " with -skipcrccheck.\n")
              .append(" (NOTE: By skipping checksums, one runs the risk of " +
                  "masking data-corruption during file-transfer.)\n");
        }
        throw new IOException(errorMessage.toString());
      }
    }
  }

  /*
   * Return the Path for a given chunk.
   * Used when splitting large file into chunks to copy in parallel.
   * @param targetFile path to target file
   * @param srcFileStatus source file status in copy listing
   * @return path to the chunk specified by the parameters to store
   * in target cluster temporarily
   */
  public static Path getSplitChunkPath(Path targetFile,
      CopyListingFileStatus srcFileStatus) {
    return new Path(targetFile.toString()
        + ".____distcpSplit____" + srcFileStatus.getChunkOffset()
        + "." + srcFileStatus.getChunkLength());
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Make checksums comparable at file level: -Ddfs.checksum.combine.mode=COMPOSITE_CRC
2. If the integrity trade-off is acceptable, disable verification with -skipcrccheck (the message warns this can mask corruption)
3. If real corruption is suspected, verify the source with 'hdfs fsck <file> -files -blocks' and recopy
4. Align dfs.block.size between source and target so checksums compare equal

Example fix

# before: per-block CRCs incomparable across filesystems -> Checksum mismatch
hadoop distcp -update hdfs://nnA/src hdfs://nnB/dst

# after (pick one)
hadoop distcp -update -Ddfs.checksum.combine.mode=COMPOSITE_CRC hdfs://nnA/src hdfs://nnB/dst
hadoop distcp -update -skipcrccheck hdfs://nnA/src hdfs://nnB/dst

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Checksum mismatch")) {
    // rerun with -Ddfs.checksum.combine.mode=COMPOSITE_CRC, or -skipcrccheck if risk accepted
  }
}

When it happens

Trigger: Source and target use different checksum types because block sizes or filesystem implementations differ (HDFS vs S3A, different dfs.block.size); actual corruption during transfer; the source file modified during the copy; clusters defaulting to different CRC algorithms.

Common situations: Cross-cluster or HDFS-to-object-store distcp where per-block CRCs are not comparable; migrations between HDFS versions with different checksum defaults; rare bit-rot during transfer.

Related errors

• Mismatch in length of source:{source} ({srcLen}) and target:
• key + ": No such file or directory."
• part etag mismatched: %s != %s
• parts length mismatched: %d != %d
• part num mismatched: %d != %d

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a39c0a1a3edd7660. Report an issue: GitHub.