{"record":{"id":"7588f99fbc0ba0ce","repo":"apache/hadoop","slug":"invalid-input-there-are-duplicated-files-in-the-s","errorCode":null,"errorMessage":"Invalid input, there are duplicated files in the sources: {prevsrc}, {cursrc}","messagePattern":"Invalid input, there are duplicated files in the sources: (.+?), (.+?)","errorType":"validation","errorClass":"DuplicationException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCh.java","lineNumber":496,"sourceCode":"    checkDuplication(fs, opList, new Path(jobdir, \"_sorted\"), jobconf);\n    jobconf.setInt(OP_COUNT_LABEL, opCount);\n    LOG.info(OP_COUNT_LABEL + \"=\" + opCount);\n    jobconf.setNumMapTasks(getMapCount(opCount,\n        new JobClient(jobconf).getClusterStatus().getTaskTrackers()));\n    return opCount != 0;    \n  }\n\n  private static void checkDuplication(FileSystem fs, Path file, Path sorted,\n    Configuration conf) throws IOException {\n    SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs,\n        new Text.Comparator(), Text.class, FileOperation.class, conf);\n    sorter.sort(file, sorted);\n    try (SequenceFile.Reader in = new SequenceFile.Reader(fs, sorted, conf)) {\n      FileOperation curop = new FileOperation();\n      Text prevsrc = null, cursrc = new Text(); \n      for(; in.next(cursrc, curop); ) {\n        if (prevsrc != null && cursrc.equals(prevsrc)) {\n          throw new DuplicationException(\n            \"Invalid input, there are duplicated files in the sources: \"\n            + prevsrc + \", \" + cursrc);\n        }\n        prevsrc = cursrc;\n        cursrc = new Text();\n        curop = new FileOperation();\n      }\n    }\n  } \n\n  public static void main(String[] args) throws Exception {\n    System.exit(ToolRunner.run(new DistCh(new Configuration()), args));\n  }\n}","sourceCodeStart":478,"sourceCodeEnd":510,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCh.java#L478-L510","documentation":"Thrown by DistCh (hadoop-extras) inside checkDuplication(), which sorts the sequence file of pending FileOperations by source path and rejects the run when two consecutive entries have the same source. It means the input listing given to DistCh contains the same source path more than once, so the chmod/chown operations would be ambiguous. The tool aborts the whole job instead of guessing which operation wins.","triggerScenarios":"Running org.apache.hadoop.tools.DistCh with a command/input file that lists the same file or directory twice. After sorter.sort() the duplicate Text keys become adjacent, and cursrc.equals(prevsrc) trips the DuplicationException. Typical sources: an input listing built from overlapping globs (e.g. '/dir /*' and '/dir/sub /*' both matching), or concatenated 'hadoop fs -ls' outputs that were never deduplicated.","commonSituations":"Scripts that concatenate multiple find/ls outputs into one DistCp input file; input lists expanded from overlapping glob patterns; a generator that appends to the listing on each run instead of overwriting it; hand-edited operation lists with copy-paste duplicates.","solutions":["Deduplicate the input listing by source path (e.g. 'sort -u' on the path column) and rerun DistCh.","If the listing is generated from globs, make the glob patterns disjoint so no path matches twice.","Regenerate the listing from a single 'hadoop fs -ls -R' pass instead of concatenating several sources.","If you genuinely need two operations on one source, split them into separate DistCh invocations."],"exampleFix":"# before: duplicates in the listing file\nhadoop fs -ls -R /data/* | awk '{print $NF}' > /tmp/chmods.list\ncat /tmp/extra.list >> /tmp/chmods.list   # may re-add paths\n\n# after: dedupe by path before running DistCh\nawk '{print $NF}' /tmp/chmods.list | sort -u > /tmp/chmods.uniq\nhadoop distch -i /tmp/chmods.uniq ...","handlingStrategy":"validation","validationCode":"// Deduplicate the DistCh input listing by source path before running the tool\nSet<String> seen = new HashSet<>();\nList<String> lines = Files.readAllLines(Paths.get(\"/tmp/chmods.list\"));\nList<String> uniq = new ArrayList<>();\nfor (String l : lines) {\n  String src = l.trim();\n  if (seen.add(src)) {\n    uniq.add(l);\n  } else {\n    LOG.warn(\"Dropping duplicate source: {}\", src);\n  }\n}\nFiles.write(Paths.get(\"/tmp/chmods.uniq\"), uniq);","typeGuard":null,"tryCatchPattern":"try {\n  DistCh.main(args);\n} catch (DuplicationException e) {\n  // listing contains repeated sources; dedupe and rerun with the file named in the message\n}","preventionTips":["Always generate DistCh listings with 'sort -u' or a Set keyed on the source path.","Use disjoint glob patterns when expanding sources into a listing.","Write listing generators in overwrite mode, never append mode."],"tags":["distch","hadoop-extras","duplicate-source","input-validation","sequence-file"],"backgroundTag":"duplicate-source-paths","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}