apache/hadoop · error · HadoopIllegalArgumentException

"concat: the src file " + src + " is the same with the targe

Error message

"concat: the src file " + src + " is the same with the target file " + targetIIP.getPath()

What it means

The src array contained the target itself: verifySrcFiles compares srcINode.equals(targetINode) and rejects it. Concat appends each source's blocks onto the target and then deletes the sources; using the target as its own source is logically undefined (a file cannot absorb itself). Note the comparison is on INode identity, so an equivalent path, a snapshot path, or a different string resolving to the same inode also triggers it.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:149

      if (srcINodeFile.getParent() != targetParent) {
        throw new HadoopIllegalArgumentException("Source file " + src
            + " is not in the same directory with the target "
            + targetIIP.getPath());
      }
      // make sure all the source files are not in snapshot
      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {
        throw new SnapshotException("Concat: the source file " + src
            + " is in snapshot");
      }
      // check if the file has other references.
      if (srcINode.isReference() && ((INodeReference.WithCount)
          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {
        throw new SnapshotException("Concat: the source file " + src
            + " is referred by some other reference in some snapshot.");
      }
      // source file cannot be the same with the target file
      if (srcINode.equals(targetINode)) {
        throw new HadoopIllegalArgumentException("concat: the src file " + src
            + " is the same with the target file " + targetIIP.getPath());
      }
      // source file cannot be under construction or empty
      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " is invalid or empty or underConstruction");
      }

      // source file's preferred block size cannot be greater than the target
      // file
      if (srcINodeFile.getPreferredBlockSize() >
          targetINode.getPreferredBlockSize()) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " has preferred block size " + srcINodeFile.getPreferredBlockSize()
            + " which is greater than the target file's preferred block size "
            + targetINode.getPreferredBlockSize());
      }
      if(srcINodeFile.getErasureCodingPolicyID() !=

View on GitHub (pinned to 2add963021)

Solutions

Filter the target out of the src list before calling concat: srcs = list minus target (compare normalized absolute paths).
Dedupe and normalize all paths (Path.makeQualified + toUri().getPath()) so aliases of the target are caught too.
If the intent is 'merge everything', pick the oldest file as target and concat only the rest.

Example fix

// before
List<Path> all = listFiles(dir); // includes target
fs.concat(target, all.toArray(new Path[0])); // target in srcs -> error

// after
String t = target.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toUri().getPath();
Path[] srcs = all.stream()
    .filter(p -> !p.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toUri().getPath().equals(t))
    .toArray(Path[]::new);
fs.concat(target, srcs);

Defensive patterns

Strategy: validation

Validate before calling

String tq = target.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toUri().getPath();
Path[] safe = Arrays.stream(srcs)
    .map(p -> p.makeQualified(fs.getUri(), fs.getWorkingDirectory()))
    .filter(p -> !p.toUri().getPath().equals(tq))
    .distinct()
    .toArray(Path[]::new);

Try / catch

catch (HadoopIllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("same with the target file")) {
    // filter target alias out of srcs and retry once
    fs.concat(target, withoutTargetAliases(target, srcs));
  } else { throw e; }
}

Prevention

When building 'merge all files in a directory' lists, always exclude the chosen target explicitly.
Normalize paths (makeQualified) before comparing so '.', double slashes, and URI-qualified aliases cannot slip through.
Unit-test compaction helpers with a src list that intentionally contains the target.

When it happens

Trigger: Passing the target path (or any alias of it, such as '/dir/f' vs '/dir/./f', or a snapshot path resolving to the same inode) in the srcs array — often from a glob that includes the target or from not filtering the target out of a directory listing.

Common situations: Compaction code that lists a directory and concats 'all files in the directory' without excluding the chosen target; glob patterns like part-* that match the target too; config-driven file lists that accidentally repeat the target.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/367194e3a8aea5cf. Report an issue: GitHub.