apache/hadoop · error · IOException

Mismatch in length of source:{source} ({srcLen}) and target:

Error message

Mismatch in length of source:{source} ({srcLen}) and target:{target} ({targetLen})

What it means

Post-copy verification in DistCpUtils.compareFileLengthsAndChecksums(): after a copy (typically -update/-overwrite), the source length and target length differ, so the copy is judged incomplete or the file changed underneath the job. The message (LENGTH_MISMATCH_ERROR_MSG prefix) names both paths with their byte lengths.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java:619

  /**
   * Utility to compare file lengths and checksums for source and target.
   *
   * @param sourceFS FileSystem for the source path.
   * @param source The source path.
   * @param sourceChecksum The checksum of the source file. If it is null we
   * still need to retrieve it through sourceFS.
   * @param targetFS FileSystem for the target path.
   * @param target The target path.
   * @param skipCrc The flag to indicate whether to skip checksums.
   * @throws IOException if there's a mismatch in file lengths or checksums.
   */
  public static void compareFileLengthsAndChecksums(long srcLen,
             FileSystem sourceFS, Path source, FileChecksum sourceChecksum,
             FileSystem targetFS, Path target, boolean skipCrc,
             long targetLen) throws IOException {
    if (srcLen != targetLen) {
      throw new IOException(
          DistCpConstants.LENGTH_MISMATCH_ERROR_MSG + source + " (" + srcLen
              + ") and target:" + target + " (" + targetLen + ")");
    }

    //At this point, src & dest lengths are same. if length==0, we skip checksum
    if ((srcLen != 0) && (!skipCrc)) {
      CopyMapper.ChecksumComparison
          checksumComparison = checksumsAreEqual(sourceFS, source, sourceChecksum,
              targetFS, target, srcLen);
      // If Checksum comparison is false set it to false, else set to true.
      boolean checksumResult = !checksumComparison.equals(CopyMapper.ChecksumComparison.FALSE);
      if (!checksumResult) {
        StringBuilder errorMessage =
            new StringBuilder(DistCpConstants.CHECKSUM_MISMATCH_ERROR_MSG)
                .append(source).append(" and ").append(target).append(".");
        boolean addSkipHint = false;
        String srcScheme = sourceFS.getScheme();
        String targetScheme = targetFS.getScheme();

View on GitHub (pinned to 2add963021)

Solutions

1. Quiesce the source (finish or pause writers) and re-run with -update to recopy the mismatched files
2. Ensure exactly one job writes a given target path at a time
3. Check target filesystem health, free space and quota
4. Compare both sides manually (hdfs dfs -duh on source and target) to see which changed

catch (IOException e) {  // 'Mismatch in length of source:... and target:...'
  // log the file pair; after quiescing writers, re-run distcp -update
}

When it happens

Trigger: The source file was appended to or truncated while being copied; the target file was written concurrently by another job; a short write due to disk or quota problems; retry resumption writing at wrong offsets.

Common situations: DistCpping directories that are still being written by live applications; two distcp jobs sharing one destination; target disk filling mid-write; -append used between files of different sizes.

Related errors

• Checksum mismatch between {} and {}.
• key + ": No such file or directory."
• parts length mismatched: %d != %d
• part num mismatched: %d != %d
• part size mismatched: %d != %d

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4fc3bd3329b8a13b. Report an issue: GitHub.