apache/hadoop · error · HadoopIllegalArgumentException

"Source file " + src + " is not in the same directory with t

Error message

"Source file " + src + " is not in the same directory with the target " + targetIIP.getPath()

What it means

HDFS concat only merges files that live in the same directory as the target: verifySrcFiles compares each source's parent INodeDirectory against the target's parent by identity and throws HadoopIllegalArgumentException on mismatch. Concat does not copy blocks — it re-splices inode block lists, which the NameNode only supports between siblings. This check runs after per-source permission checks (READ on the file, WRITE on the parent).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:132

  private static INodeFile[] verifySrcFiles(FSDirectory fsd, String[] srcs,
      INodesInPath targetIIP, FSPermissionChecker pc) throws IOException {
    // to make sure no two files are the same
    Set<INodeFile> si = new LinkedHashSet<>();
    final INodeFile targetINode = targetIIP.getLastINode().asFile();
    final INodeDirectory targetParent = targetINode.getParent();
    // now check the srcs
    for(String src : srcs) {
      final INodesInPath iip = fsd.resolvePath(pc, src, DirOp.WRITE);
      // permission check for srcs
      if (pc != null && fsd.isPermissionEnabled()) {
        fsd.checkPathAccess(pc, iip, FsAction.READ); // read the file
        fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete
      }
      final INode srcINode = iip.getLastINode();
      final INodeFile srcINodeFile = INodeFile.valueOf(srcINode, src);
      // make sure the src file and the target file are in the same dir
      if (srcINodeFile.getParent() != targetParent) {
        throw new HadoopIllegalArgumentException("Source file " + src
            + " is not in the same directory with the target "
            + targetIIP.getPath());
      }
      // make sure all the source files are not in snapshot
      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {
        throw new SnapshotException("Concat: the source file " + src
            + " is in snapshot");
      }
      // check if the file has other references.
      if (srcINode.isReference() && ((INodeReference.WithCount)
          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {
        throw new SnapshotException("Concat: the source file " + src
            + " is referred by some other reference in some snapshot.");
      }
      // source file cannot be the same with the target file
      if (srcINode.equals(targetINode)) {
        throw new HadoopIllegalArgumentException("concat: the src file " + src
            + " is the same with the target file " + targetIIP.getPath());

View on GitHub (pinned to 2add963021)

Solutions

Group source files by parent directory and issue one concat per directory with that directory's target.
If sources live elsewhere, rename them into the target's directory first (fs.rename) and then concat.
Validate src.getParent().equals(target.getParent()) for every src before calling concat.

Example fix

// before
fs.concat(new Path("/data/final/part-0000"),
         new Path[]{new Path("/data/final/part-0001"),
                    new Path("/data/tmp/part-0002")}); // /data/tmp src -> error

// after
Path tmp = new Path("/data/tmp/part-0002");
fs.rename(tmp, new Path("/data/final/part-0002"));
fs.concat(new Path("/data/final/part-0000"),
         new Path[]{new Path("/data/final/part-0001"),
                    new Path("/data/final/part-0002")});

Defensive patterns

Strategy: validation

Validate before calling

Path targetParent = target.getParent();
for (Path src : srcs) {
  if (!targetParent.equals(src.getParent())) {
    throw new IllegalArgumentException(src + " not in same directory as " + target);
  }
}

Try / catch

catch (HadoopIllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("same directory")) {
    // group srcs by parent and concat per group instead of failing the batch
    concatPerDirectory(target.getParent().getParent(), srcs);
  } else { throw e; }
}

Prevention

Group candidate files by parent directory during planning; never build cross-directory concat batches.
If sources live in a staging dir, rename them next to the target before concat.
Assert src.getParent().equals(target.getParent()) in tests for compaction utilities.

When it happens

Trigger: FileSystem.concat(target, srcs) where at least one src path resolves to a file whose parent directory differs from the target's, even by one level (e.g. /data/final and /data/tmp).

Common situations: Compaction jobs that glob across a tree (e.g. /data/*/part-*) and pass everything to one target; temp-directory patterns where writers land files in /tmp and the target lives in /data; porting local-filesystem concat logic where any paths are accepted.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4d5ac53d746fb5eb. Report an issue: GitHub.