apache/hadoop · error · DuplicationException

Invalid input, there are duplicated files in the sources: {p

Error message

Invalid input, there are duplicated files in the sources: {prevsrc}, {cursrc}

What it means

Thrown by DistCh (hadoop-extras) inside checkDuplication(), which sorts the sequence file of pending FileOperations by source path and rejects the run when two consecutive entries have the same source. It means the input listing given to DistCh contains the same source path more than once, so the chmod/chown operations would be ambiguous. The tool aborts the whole job instead of guessing which operation wins.

Source

Thrown at hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCh.java:496

    checkDuplication(fs, opList, new Path(jobdir, "_sorted"), jobconf);
    jobconf.setInt(OP_COUNT_LABEL, opCount);
    LOG.info(OP_COUNT_LABEL + "=" + opCount);
    jobconf.setNumMapTasks(getMapCount(opCount,
        new JobClient(jobconf).getClusterStatus().getTaskTrackers()));
    return opCount != 0;    
  }

  private static void checkDuplication(FileSystem fs, Path file, Path sorted,
    Configuration conf) throws IOException {
    SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs,
        new Text.Comparator(), Text.class, FileOperation.class, conf);
    sorter.sort(file, sorted);
    try (SequenceFile.Reader in = new SequenceFile.Reader(fs, sorted, conf)) {
      FileOperation curop = new FileOperation();
      Text prevsrc = null, cursrc = new Text(); 
      for(; in.next(cursrc, curop); ) {
        if (prevsrc != null && cursrc.equals(prevsrc)) {
          throw new DuplicationException(
            "Invalid input, there are duplicated files in the sources: "
            + prevsrc + ", " + cursrc);
        }
        prevsrc = cursrc;
        cursrc = new Text();
        curop = new FileOperation();
      }
    }
  } 

  public static void main(String[] args) throws Exception {
    System.exit(ToolRunner.run(new DistCh(new Configuration()), args));
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Deduplicate the input listing by source path (e.g. 'sort -u' on the path column) and rerun DistCh.
If the listing is generated from globs, make the glob patterns disjoint so no path matches twice.
Regenerate the listing from a single 'hadoop fs -ls -R' pass instead of concatenating several sources.
If you genuinely need two operations on one source, split them into separate DistCh invocations.

Example fix

# before: duplicates in the listing file
hadoop fs -ls -R /data/* | awk '{print $NF}' > /tmp/chmods.list
cat /tmp/extra.list >> /tmp/chmods.list   # may re-add paths

# after: dedupe by path before running DistCh
awk '{print $NF}' /tmp/chmods.list | sort -u > /tmp/chmods.uniq
hadoop distch -i /tmp/chmods.uniq ...

Defensive patterns

Strategy: validation

Validate before calling

// Deduplicate the DistCh input listing by source path before running the tool
Set<String> seen = new HashSet<>();
List<String> lines = Files.readAllLines(Paths.get("/tmp/chmods.list"));
List<String> uniq = new ArrayList<>();
for (String l : lines) {
  String src = l.trim();
  if (seen.add(src)) {
    uniq.add(l);
  } else {
    LOG.warn("Dropping duplicate source: {}", src);
  }
}
Files.write(Paths.get("/tmp/chmods.uniq"), uniq);

Try / catch

try {
  DistCh.main(args);
} catch (DuplicationException e) {
  // listing contains repeated sources; dedupe and rerun with the file named in the message
}

Prevention

Always generate DistCh listings with 'sort -u' or a Set keyed on the source path.
Use disjoint glob patterns when expanding sources into a listing.
Write listing generators in overwrite mode, never append mode.

When it happens

Trigger: Running org.apache.hadoop.tools.DistCh with a command/input file that lists the same file or directory twice. After sorter.sort() the duplicate Text keys become adjacent, and cursrc.equals(prevsrc) trips the DuplicationException. Typical sources: an input listing built from overlapping globs (e.g. '/dir /*' and '/dir/sub /*' both matching), or concatenated 'hadoop fs -ls' outputs that were never deduplicated.

Common situations: Scripts that concatenate multiple find/ls outputs into one DistCp input file; input lists expanded from overlapping glob patterns; a generator that appends to the listing on each run instead of overwriting it; hand-edited operation lists with copy-paste duplicates.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7588f99fbc0ba0ce. Report an issue: GitHub.