{"record":{"id":"4d5ac53d746fb5eb","repo":"apache/hadoop","slug":"source-file-src-is-not-in-the-same-direct","errorCode":null,"errorMessage":"\"Source file \" + src + \" is not in the same directory with the target \" + targetIIP.getPath()","messagePattern":"\"Source file \" \\+ src \\+ \" is not in the same directory with the target \" \\+ targetIIP\\.getPath\\(\\)","errorType":"exception","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":132,"sourceCode":"  private static INodeFile[] verifySrcFiles(FSDirectory fsd, String[] srcs,\n      INodesInPath targetIIP, FSPermissionChecker pc) throws IOException {\n    // to make sure no two files are the same\n    Set<INodeFile> si = new LinkedHashSet<>();\n    final INodeFile targetINode = targetIIP.getLastINode().asFile();\n    final INodeDirectory targetParent = targetINode.getParent();\n    // now check the srcs\n    for(String src : srcs) {\n      final INodesInPath iip = fsd.resolvePath(pc, src, DirOp.WRITE);\n      // permission check for srcs\n      if (pc != null && fsd.isPermissionEnabled()) {\n        fsd.checkPathAccess(pc, iip, FsAction.READ); // read the file\n        fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete\n      }\n      final INode srcINode = iip.getLastINode();\n      final INodeFile srcINodeFile = INodeFile.valueOf(srcINode, src);\n      // make sure the src file and the target file are in the same dir\n      if (srcINodeFile.getParent() != targetParent) {\n        throw new HadoopIllegalArgumentException(\"Source file \" + src\n            + \" is not in the same directory with the target \"\n            + targetIIP.getPath());\n      }\n      // make sure all the source files are not in snapshot\n      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is in snapshot\");\n      }\n      // check if the file has other references.\n      if (srcINode.isReference() && ((INodeReference.WithCount)\n          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is referred by some other reference in some snapshot.\");\n      }\n      // source file cannot be the same with the target file\n      if (srcINode.equals(targetINode)) {\n        throw new HadoopIllegalArgumentException(\"concat: the src file \" + src\n            + \" is the same with the target file \" + targetIIP.getPath());","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L114-L150","documentation":"HDFS concat only merges files that live in the same directory as the target: verifySrcFiles compares each source's parent INodeDirectory against the target's parent by identity and throws HadoopIllegalArgumentException on mismatch. Concat does not copy blocks — it re-splices inode block lists, which the NameNode only supports between siblings. This check runs after per-source permission checks (READ on the file, WRITE on the parent).","triggerScenarios":"FileSystem.concat(target, srcs) where at least one src path resolves to a file whose parent directory differs from the target's, even by one level (e.g. /data/final and /data/tmp).","commonSituations":"Compaction jobs that glob across a tree (e.g. /data/*/part-*) and pass everything to one target; temp-directory patterns where writers land files in /tmp and the target lives in /data; porting local-filesystem concat logic where any paths are accepted.","solutions":["Group source files by parent directory and issue one concat per directory with that directory's target.","If sources live elsewhere, rename them into the target's directory first (fs.rename) and then concat.","Validate src.getParent().equals(target.getParent()) for every src before calling concat."],"exampleFix":"// before\nfs.concat(new Path(\"/data/final/part-0000\"),\n         new Path[]{new Path(\"/data/final/part-0001\"),\n                    new Path(\"/data/tmp/part-0002\")}); // /data/tmp src -> error\n\n// after\nPath tmp = new Path(\"/data/tmp/part-0002\");\nfs.rename(tmp, new Path(\"/data/final/part-0002\"));\nfs.concat(new Path(\"/data/final/part-0000\"),\n         new Path[]{new Path(\"/data/final/part-0001\"),\n                    new Path(\"/data/final/part-0002\")});","handlingStrategy":"validation","validationCode":"Path targetParent = target.getParent();\nfor (Path src : srcs) {\n  if (!targetParent.equals(src.getParent())) {\n    throw new IllegalArgumentException(src + \" not in same directory as \" + target);\n  }\n}","typeGuard":null,"tryCatchPattern":"catch (HadoopIllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"same directory\")) {\n    // group srcs by parent and concat per group instead of failing the batch\n    concatPerDirectory(target.getParent().getParent(), srcs);\n  } else { throw e; }\n}","preventionTips":["Group candidate files by parent directory during planning; never build cross-directory concat batches.","If sources live in a staging dir, rename them next to the target before concat.","Assert src.getParent().equals(target.getParent()) in tests for compaction utilities."],"tags":["hdfs","concat","same-directory","precondition"],"backgroundTag":"hdfs-concat-precondition","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}