apache/hadoop · error · DuplicateFileException

File " + lastFileStatus.getPath() + " and " + currentFileSta

Error message

File " + lastFileStatus.getPath() + " and " + currentFileStatus.getPath() + " would cause duplicates. Aborting

What it means

CopyListing validates the generated sequence file of files to copy (sorted by relative-path key). Two consecutive entries with the same key mean two different source files would be written to the same target path — the copy would silently overwrite one of them. DistCp aborts up front with DuplicateFileException('File X and Y would cause duplicates. Aborting') instead; this only happens when splitLargeFile (-blocksperchunk) is off.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java:174

    SequenceFile.Reader reader = new SequenceFile.Reader(
                          config, SequenceFile.Reader.file(checkPath));
    try {
      Text lastKey = new Text("*"); //source relative path can never hold *
      long lastChunkOffset = -1;
      long lastChunkLength = -1;
      CopyListingFileStatus lastFileStatus = new CopyListingFileStatus();

      Text currentKey = new Text();
      Set<URI> aclSupportCheckFsSet = Sets.newHashSet();
      Set<URI> xAttrSupportCheckFsSet = Sets.newHashSet();
      long idx = 0;
      while (reader.next(currentKey)) {
        if (currentKey.equals(lastKey)) {
          CopyListingFileStatus currentFileStatus = new CopyListingFileStatus();
          reader.getCurrentValue(currentFileStatus);
          if (!splitLargeFile) {
            throw new DuplicateFileException("File " + lastFileStatus.getPath()
                + " and " + currentFileStatus.getPath()
                + " would cause duplicates. Aborting");
          } else {
            if (lastChunkOffset + lastChunkLength !=
                currentFileStatus.getChunkOffset()) {
              throw new InvalidInputException("File " + lastFileStatus.getPath()
                  + " " + lastChunkOffset + "," + lastChunkLength
                  + " and " + currentFileStatus.getPath()
                  + " " + currentFileStatus.getChunkOffset() + ","
                  + currentFileStatus.getChunkLength()
                  + " are not continuous. Aborting");
            }
          }
        }
        reader.getCurrentValue(lastFileStatus);
        if (context.shouldPreserve(DistCpOptions.FileAttribute.ACL)) {
          FileSystem lastFs = lastFileStatus.getPath().getFileSystem(config);
          URI lastFsUri = lastFs.getUri();

View on GitHub (pinned to 2add963021)

Solutions

Remove overlapping sources: drop the child path when its parent is already included
Deduplicate the -f listing before running: hadoop fs -cat listing.txt | sort -u > listing.dedup.txt
If the collision is legitimate (same relative name from different roots), run separate distcp commands into different target subdirectories
For case-collisions, rename one of the conflicting sources

Example fix

# before: listing.txt contains /data/a/part-0 twice (or overlapping dirs)
hadoop distcp -f listing.txt hdfs://nn/dst
# -> File /data/a/part-0 and /data/a/part-0 would cause duplicates. Aborting

# after
hadoop fs -cat listing.txt | sort -u > /tmp/listing.dedup.txt
hadoop distcp -f /tmp/listing.dedup.txt hdfs://nn/dst

Defensive patterns

Strategy: validation

Validate before calling

// reject duplicate and overlapping sources before building the listing
Set<Path> qualified = new LinkedHashSet<>();
for (Path src : sourcePaths) {
  if (!qualified.add(src.makeQualified(fs.getUri(), fs.getWorkingDir()))) {
    throw new IllegalArgumentException("duplicate source path: " + src);
  }
}
for (Path a : qualified) {
  for (Path b : qualified) {
    if (!a.equals(b) && isUnder(b, a)) { // b is a descendant of a
      throw new IllegalArgumentException(
          "overlapping sources: " + a + " already contains " + b);
    }
  }
}

Try / catch

DuplicateFileException is a package-private RuntimeException: catch RuntimeException around DistCp.execute(), match 'would cause duplicates' in the message, print the two colliding paths, then dedupe the inputs (sort -u the listing or drop overlapping dirs) and rerun.

Prevention

Generate -f listings with sort -u and review them before submission
Never pass a directory together with its own subdirectory as distcp sources
When merging listings from multiple roots, check relative-path collisions against the target first

When it happens

Trigger: Overlapping source arguments (copying /data and /data/sub in one command yields the child's files twice); a file-based listing (-f) containing the same path twice; two sources whose relative paths collide on the target (both trees contain part-0, or names differing only in letter case).

Common situations: -f listings generated by concatenating multiple find outputs without dedup; mixing snapshot roots or different source roots with identical layout; case-insensitive source filesystems producing names that collide on a case-sensitive target; passing a directory and its own subdirectory together.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9e5bd366fdbc7264. Report an issue: GitHub.