{"record":{"id":"4fc3bd3329b8a13b","repo":"apache/hadoop","slug":"mismatch-in-length-of-source-source-srclen-a","errorCode":null,"errorMessage":"Mismatch in length of source:{source} ({srcLen}) and target:{target} ({targetLen})","messagePattern":"Mismatch in length of source:(.+?) \\((.+?)\\) and target:(.+?) \\((.+?)\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java","lineNumber":619,"sourceCode":"\n  /**\n   * Utility to compare file lengths and checksums for source and target.\n   *\n   * @param sourceFS FileSystem for the source path.\n   * @param source The source path.\n   * @param sourceChecksum The checksum of the source file. If it is null we\n   * still need to retrieve it through sourceFS.\n   * @param targetFS FileSystem for the target path.\n   * @param target The target path.\n   * @param skipCrc The flag to indicate whether to skip checksums.\n   * @throws IOException if there's a mismatch in file lengths or checksums.\n   */\n  public static void compareFileLengthsAndChecksums(long srcLen,\n             FileSystem sourceFS, Path source, FileChecksum sourceChecksum,\n             FileSystem targetFS, Path target, boolean skipCrc,\n             long targetLen) throws IOException {\n    if (srcLen != targetLen) {\n      throw new IOException(\n          DistCpConstants.LENGTH_MISMATCH_ERROR_MSG + source + \" (\" + srcLen\n              + \") and target:\" + target + \" (\" + targetLen + \")\");\n    }\n\n    //At this point, src & dest lengths are same. if length==0, we skip checksum\n    if ((srcLen != 0) && (!skipCrc)) {\n      CopyMapper.ChecksumComparison\n          checksumComparison = checksumsAreEqual(sourceFS, source, sourceChecksum,\n              targetFS, target, srcLen);\n      // If Checksum comparison is false set it to false, else set to true.\n      boolean checksumResult = !checksumComparison.equals(CopyMapper.ChecksumComparison.FALSE);\n      if (!checksumResult) {\n        StringBuilder errorMessage =\n            new StringBuilder(DistCpConstants.CHECKSUM_MISMATCH_ERROR_MSG)\n                .append(source).append(\" and \").append(target).append(\".\");\n        boolean addSkipHint = false;\n        String srcScheme = sourceFS.getScheme();\n        String targetScheme = targetFS.getScheme();","sourceCodeStart":601,"sourceCodeEnd":637,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java#L601-L637","documentation":"Post-copy verification in DistCpUtils.compareFileLengthsAndChecksums(): after a copy (typically -update/-overwrite), the source length and target length differ, so the copy is judged incomplete or the file changed underneath the job. The message (LENGTH_MISMATCH_ERROR_MSG prefix) names both paths with their byte lengths.","triggerScenarios":"The source file was appended to or truncated while being copied; the target file was written concurrently by another job; a short write due to disk or quota problems; retry resumption writing at wrong offsets.","commonSituations":"DistCpping directories that are still being written by live applications; two distcp jobs sharing one destination; target disk filling mid-write; -append used between files of different sizes.","solutions":["Quiesce the source (finish or pause writers) and re-run with -update to recopy the mismatched files","Ensure exactly one job writes a given target path at a time","Check target filesystem health, free space and quota","Compare both sides manually (hdfs dfs -duh on source and target) to see which changed"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) {  // 'Mismatch in length of source:... and target:...'\n  // log the file pair; after quiescing writers, re-run distcp -update\n}","preventionTips":["Copy from quiesced sources; pause writers during large distcp runs","One job per target path at a time","Watch target disk space and quota during long copies","Re-run with -update instead of full recopies"],"tags":["distcp","hadoop","verification","length-mismatch","data-integrity"],"backgroundTag":"data-integrity-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}