{"record":{"id":"9e5bd366fdbc7264","repo":"apache/hadoop","slug":"file-lastfilestatus-getpath-and-curr","errorCode":null,"errorMessage":"File \" + lastFileStatus.getPath() + \" and \" + currentFileStatus.getPath() + \" would cause duplicates. Aborting","messagePattern":"File \" \\+ lastFileStatus\\.getPath\\(\\) \\+ \" and \" \\+ currentFileStatus\\.getPath\\(\\) \\+ \" would cause duplicates\\. Aborting","errorType":"exception","errorClass":"DuplicateFileException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java","lineNumber":174,"sourceCode":"\n    SequenceFile.Reader reader = new SequenceFile.Reader(\n                          config, SequenceFile.Reader.file(checkPath));\n    try {\n      Text lastKey = new Text(\"*\"); //source relative path can never hold *\n      long lastChunkOffset = -1;\n      long lastChunkLength = -1;\n      CopyListingFileStatus lastFileStatus = new CopyListingFileStatus();\n\n      Text currentKey = new Text();\n      Set<URI> aclSupportCheckFsSet = Sets.newHashSet();\n      Set<URI> xAttrSupportCheckFsSet = Sets.newHashSet();\n      long idx = 0;\n      while (reader.next(currentKey)) {\n        if (currentKey.equals(lastKey)) {\n          CopyListingFileStatus currentFileStatus = new CopyListingFileStatus();\n          reader.getCurrentValue(currentFileStatus);\n          if (!splitLargeFile) {\n            throw new DuplicateFileException(\"File \" + lastFileStatus.getPath()\n                + \" and \" + currentFileStatus.getPath()\n                + \" would cause duplicates. Aborting\");\n          } else {\n            if (lastChunkOffset + lastChunkLength !=\n                currentFileStatus.getChunkOffset()) {\n              throw new InvalidInputException(\"File \" + lastFileStatus.getPath()\n                  + \" \" + lastChunkOffset + \",\" + lastChunkLength\n                  + \" and \" + currentFileStatus.getPath()\n                  + \" \" + currentFileStatus.getChunkOffset() + \",\"\n                  + currentFileStatus.getChunkLength()\n                  + \" are not continuous. Aborting\");\n            }\n          }\n        }\n        reader.getCurrentValue(lastFileStatus);\n        if (context.shouldPreserve(DistCpOptions.FileAttribute.ACL)) {\n          FileSystem lastFs = lastFileStatus.getPath().getFileSystem(config);\n          URI lastFsUri = lastFs.getUri();","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java#L156-L192","documentation":"CopyListing validates the generated sequence file of files to copy (sorted by relative-path key). Two consecutive entries with the same key mean two different source files would be written to the same target path — the copy would silently overwrite one of them. DistCp aborts up front with DuplicateFileException('File X and Y would cause duplicates. Aborting') instead; this only happens when splitLargeFile (-blocksperchunk) is off.","triggerScenarios":"Overlapping source arguments (copying /data and /data/sub in one command yields the child's files twice); a file-based listing (-f) containing the same path twice; two sources whose relative paths collide on the target (both trees contain part-0, or names differing only in letter case).","commonSituations":"-f listings generated by concatenating multiple find outputs without dedup; mixing snapshot roots or different source roots with identical layout; case-insensitive source filesystems producing names that collide on a case-sensitive target; passing a directory and its own subdirectory together.","solutions":["Remove overlapping sources: drop the child path when its parent is already included","Deduplicate the -f listing before running: hadoop fs -cat listing.txt | sort -u > listing.dedup.txt","If the collision is legitimate (same relative name from different roots), run separate distcp commands into different target subdirectories","For case-collisions, rename one of the conflicting sources"],"exampleFix":"# before: listing.txt contains /data/a/part-0 twice (or overlapping dirs)\nhadoop distcp -f listing.txt hdfs://nn/dst\n# -> File /data/a/part-0 and /data/a/part-0 would cause duplicates. Aborting\n\n# after\nhadoop fs -cat listing.txt | sort -u > /tmp/listing.dedup.txt\nhadoop distcp -f /tmp/listing.dedup.txt hdfs://nn/dst","handlingStrategy":"validation","validationCode":"// reject duplicate and overlapping sources before building the listing\nSet<Path> qualified = new LinkedHashSet<>();\nfor (Path src : sourcePaths) {\n  if (!qualified.add(src.makeQualified(fs.getUri(), fs.getWorkingDir()))) {\n    throw new IllegalArgumentException(\"duplicate source path: \" + src);\n  }\n}\nfor (Path a : qualified) {\n  for (Path b : qualified) {\n    if (!a.equals(b) && isUnder(b, a)) { // b is a descendant of a\n      throw new IllegalArgumentException(\n          \"overlapping sources: \" + a + \" already contains \" + b);\n    }\n  }\n}","typeGuard":null,"tryCatchPattern":"DuplicateFileException is a package-private RuntimeException: catch RuntimeException around DistCp.execute(), match 'would cause duplicates' in the message, print the two colliding paths, then dedupe the inputs (sort -u the listing or drop overlapping dirs) and rerun.","preventionTips":["Generate -f listings with sort -u and review them before submission","Never pass a directory together with its own subdirectory as distcp sources","When merging listings from multiple roots, check relative-path collisions against the target first"],"tags":["distcp","duplicate-paths","input-validation","file-listing"],"backgroundTag":"duplicate-input-path","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}