{"record":{"id":"3565e0e628f49127","repo":"apache/hadoop","slug":"concat-at-least-two-of-the-source-files-are-the-s","errorCode":null,"errorMessage":"concat: at least two of the source files are the same","messagePattern":"concat: at least two of the source files are the same","errorType":"exception","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":179,"sourceCode":"          targetINode.getPreferredBlockSize()) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" has preferred block size \" + srcINodeFile.getPreferredBlockSize()\n            + \" which is greater than the target file's preferred block size \"\n            + targetINode.getPreferredBlockSize());\n      }\n      if(srcINodeFile.getErasureCodingPolicyID() !=\n          targetINode.getErasureCodingPolicyID()) {\n        throw new HadoopIllegalArgumentException(\"Source file \" + src\n            + \" and target file \" + targetIIP.getPath()\n            + \" have different erasure coding policy\");\n      }\n      si.add(srcINodeFile);\n    }\n\n    // make sure no two files are the same\n    if(si.size() < srcs.length) {\n      // it means at least two files are the same\n      throw new HadoopIllegalArgumentException(\n          \"concat: at least two of the source files are the same\");\n    }\n    return si.toArray(new INodeFile[si.size()]);\n  }\n\n  private static QuotaCounts computeQuotaDeltas(FSDirectory fsd,\n      INodeFile target, INodeFile[] srcList) {\n    QuotaCounts deltas = new QuotaCounts.Builder().build();\n    final short targetRepl = target.getPreferredBlockReplication();\n    for (INodeFile src : srcList) {\n      short srcRepl = src.getFileReplication();\n      long fileSize = src.computeFileSize();\n      if (targetRepl != srcRepl) {\n        deltas.addStorageSpace(fileSize * (targetRepl - srcRepl));\n        BlockStoragePolicy bsp =\n            fsd.getBlockStoragePolicySuite().getPolicy(src.getStoragePolicyID());\n        if (bsp != null) {\n          List<StorageType> srcTypeChosen = bsp.chooseStorageTypes(srcRepl);","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L161-L197","documentation":"verifySrcFiles collects sources into a LinkedHashSet keyed by INodeFile; if the set ends up smaller than the srcs array, at least two array entries resolved to the same inode and HadoopIllegalArgumentException is thrown. This is the sibling of the target-equality check: paths need not be string-equal — different path strings ( '.', snapshot aliases) resolving to the same file still count as duplicates.","triggerScenarios":"Passing the same file twice in srcs, either as an identical string or as different strings that resolve to the same inode (e.g. '/d/f' and '/d/./f', or a path plus its snapshot alias); de-duplicating a glob by filename rather than by full path.","commonSituations":"Building src lists by concatenating results of multiple globs that overlap; re-running a failed compaction round whose src manifest is merged with the next round's; user-supplied file lists containing repeats.","solutions":["Deduplicate the src array by qualified absolute path (Set<String> of makeQualified().toUri().getPath()) before calling concat.","When merging manifests across retry rounds, key by path (or inode-identifying path) and replace, not append.","Log the deduped list at DEBUG so repeated inputs are visible during compaction debugging."],"exampleFix":"// before\nList<Path> srcs = Lists.newArrayList();\nsrcs.addAll(glob(\"/data/part-*\"));\nsrcs.addAll(retryManifest); // may repeat files -> error\nfs.concat(target, srcs.toArray(new Path[0]));\n\n// after\nSet<String> seen = new HashSet<>();\nList<Path> srcs = new ArrayList<>();\nfor (Path p : Iterables.concat(glob(\"/data/part-*\"), retryManifest)) {\n  if (seen.add(p.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toUri().getPath())) {\n    srcs.add(p);\n  }\n}\nfs.concat(target, srcs.toArray(new Path[0]));","handlingStrategy":"validation","validationCode":"Set<String> seen = new HashSet<>();\nList<Path> unique = new ArrayList<>();\nfor (Path p : srcs) {\n  String k = p.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toUri().getPath();\n  if (seen.add(k)) unique.add(p);\n}","typeGuard":null,"tryCatchPattern":"catch (HadoopIllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"at least two of the source files\")) {\n    fs.concat(target, dedupeByQualifiedPath(srcs)); // retry with deduplicated list\n  } else { throw e; }\n}","preventionTips":["Deduplicate src lists by qualified absolute path, never by file name alone.","When merging retry manifests with fresh globs, key by path and replace rather than append.","Emit the final src list in job logs so duplicate inputs are diagnosable from history."],"tags":["hdfs","concat","duplicate-source","precondition"],"backgroundTag":"concat-duplicate-source","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}