What causes "concat: at least two of the source files are the same" in apache/hadoop?

Deduplicate the src array by qualified absolute path (Set of makeQualified().toUri().getPath()) before calling concat.

apache/hadoop · error · HadoopIllegalArgumentException

concat: at least two of the source files are the same

Error message

concat: at least two of the source files are the same

What it means

verifySrcFiles collects sources into a LinkedHashSet keyed by INodeFile; if the set ends up smaller than the srcs array, at least two array entries resolved to the same inode and HadoopIllegalArgumentException is thrown. This is the sibling of the target-equality check: paths need not be string-equal — different path strings ( '.', snapshot aliases) resolving to the same file still count as duplicates.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:179

          targetINode.getPreferredBlockSize()) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " has preferred block size " + srcINodeFile.getPreferredBlockSize()
            + " which is greater than the target file's preferred block size "
            + targetINode.getPreferredBlockSize());
      }
      if(srcINodeFile.getErasureCodingPolicyID() !=
          targetINode.getErasureCodingPolicyID()) {
        throw new HadoopIllegalArgumentException("Source file " + src
            + " and target file " + targetIIP.getPath()
            + " have different erasure coding policy");
      }
      si.add(srcINodeFile);
    }

    // make sure no two files are the same
    if(si.size() < srcs.length) {
      // it means at least two files are the same
      throw new HadoopIllegalArgumentException(
          "concat: at least two of the source files are the same");
    }
    return si.toArray(new INodeFile[si.size()]);
  }

  private static QuotaCounts computeQuotaDeltas(FSDirectory fsd,
      INodeFile target, INodeFile[] srcList) {
    QuotaCounts deltas = new QuotaCounts.Builder().build();
    final short targetRepl = target.getPreferredBlockReplication();
    for (INodeFile src : srcList) {
      short srcRepl = src.getFileReplication();
      long fileSize = src.computeFileSize();
      if (targetRepl != srcRepl) {
        deltas.addStorageSpace(fileSize * (targetRepl - srcRepl));
        BlockStoragePolicy bsp =
            fsd.getBlockStoragePolicySuite().getPolicy(src.getStoragePolicyID());
        if (bsp != null) {
          List<StorageType> srcTypeChosen = bsp.chooseStorageTypes(srcRepl);

View on GitHub (pinned to 2add963021)

Solutions

Deduplicate the src array by qualified absolute path (Set<String> of makeQualified().toUri().getPath()) before calling concat.
When merging manifests across retry rounds, key by path (or inode-identifying path) and replace, not append.
Log the deduped list at DEBUG so repeated inputs are visible during compaction debugging.

Example fix

// before
List<Path> srcs = Lists.newArrayList();
srcs.addAll(glob("/data/part-*"));
srcs.addAll(retryManifest); // may repeat files -> error
fs.concat(target, srcs.toArray(new Path[0]));

// after
Set<String> seen = new HashSet<>();
List<Path> srcs = new ArrayList<>();
for (Path p : Iterables.concat(glob("/data/part-*"), retryManifest)) {
  if (seen.add(p.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toUri().getPath())) {
    srcs.add(p);
  }
}
fs.concat(target, srcs.toArray(new Path[0]));

Defensive patterns

Strategy: validation

Validate before calling

Set<String> seen = new HashSet<>();
List<Path> unique = new ArrayList<>();
for (Path p : srcs) {
  String k = p.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toUri().getPath();
  if (seen.add(k)) unique.add(p);
}

Try / catch

catch (HadoopIllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("at least two of the source files")) {
    fs.concat(target, dedupeByQualifiedPath(srcs)); // retry with deduplicated list
  } else { throw e; }
}

Prevention

Deduplicate src lists by qualified absolute path, never by file name alone.
When merging retry manifests with fresh globs, key by path and replace rather than append.
Emit the final src list in job logs so duplicate inputs are diagnosable from history.

When it happens

Trigger: Passing the same file twice in srcs, either as an identical string or as different strings that resolve to the same inode (e.g. '/d/f' and '/d/./f', or a path plus its snapshot alias); de-duplicating a glob by filename rather than by full path.

Common situations: Building src lists by concatenating results of multiple globs that overlap; re-running a failed compaction round whose src manifest is merged with the next round's; user-supplied file lists containing repeats.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3565e0e628f49127. Report an issue: GitHub.