{"record":{"id":"31130f0bfdf336ce","repo":"apache/hadoop","slug":"concat-the-source-file-src-is-referred-b","errorCode":null,"errorMessage":"\"Concat: the source file \" + src + \" is referred by some other reference in some snapshot.\"","messagePattern":"\"Concat: the source file \" \\+ src \\+ \" is referred by some other reference in some snapshot\\.\"","errorType":"exception","errorClass":"SnapshotException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":144,"sourceCode":"        fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete\n      }\n      final INode srcINode = iip.getLastINode();\n      final INodeFile srcINodeFile = INodeFile.valueOf(srcINode, src);\n      // make sure the src file and the target file are in the same dir\n      if (srcINodeFile.getParent() != targetParent) {\n        throw new HadoopIllegalArgumentException(\"Source file \" + src\n            + \" is not in the same directory with the target \"\n            + targetIIP.getPath());\n      }\n      // make sure all the source files are not in snapshot\n      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is in snapshot\");\n      }\n      // check if the file has other references.\n      if (srcINode.isReference() && ((INodeReference.WithCount)\n          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is referred by some other reference in some snapshot.\");\n      }\n      // source file cannot be the same with the target file\n      if (srcINode.equals(targetINode)) {\n        throw new HadoopIllegalArgumentException(\"concat: the src file \" + src\n            + \" is the same with the target file \" + targetIIP.getPath());\n      }\n      // source file cannot be under construction or empty\n      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" is invalid or empty or underConstruction\");\n      }\n\n      // source file's preferred block size cannot be greater than the target\n      // file\n      if (srcINodeFile.getPreferredBlockSize() >\n          targetINode.getPreferredBlockSize()) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src","sourceCodeStart":126,"sourceCodeEnd":162,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L126-L162","documentation":"The source inode is an INodeReference (created by rename inside/between snapshot-captured directories) with a reference count above one, so other snapshot views still point at it. verifySrcFiles detects srcINode.isReference() with getReferenceCount() > 1 and throws SnapshotException, because consuming such a file via concat would mutate storage still visible through those other references.","triggerScenarios":"Concat on a source that was renamed while a snapshot covered the original location — the rename creates a reference pair (WithCount), leaving the file reachable from both the snapshot path and the current path.","commonSituations":"Data pipelines that rename files into a final directory that also carries snapshots (classic 'write to temp, rename to final' under snapshot); snapshot-based audit systems on directories that compaction later tries to merge.","solutions":["Delete the snapshots that hold references to the source (check hdfs lsSnapshotResults / diff reports to find which snapshot references it), dropping the reference count to 1, then concat.","Skip referenced sources in this compaction round and report them for manual handling.","Avoid renaming files within snapshot-captured directories when later concat is planned."],"exampleFix":"// before\nfs.rename(new Path(\"/data/.staging/f\"), new Path(\"/data/f\")); // /data has snapshots\nfs.concat(target, new Path[]{new Path(\"/data/f\")}); // refcount > 1 -> SnapshotException\n\n// after\n// remove snapshots of /data that reference f, then:\nfs.concat(target, new Path[]{new Path(\"/data/f\")});","handlingStrategy":"try-catch","validationCode":"// references only exist for files renamed under snapshots; avoid the pattern by writing directly\n// to the final path instead of temp-then-rename inside snapshot-captured directories.","typeGuard":null,"tryCatchPattern":"catch (SnapshotException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"referred by some other reference\")) {\n    // this src is shared with a snapshot view: skip it and surface for manual snapshot cleanup\n    skipAndReport(src);\n  } else { throw e; }\n}","preventionTips":["Do not rename files within directories covered by snapshots when the file may later be a concat source.","Detect rename-under-snapshot patterns in pipelines (temp dir -> final dir where final dir has snapshots) during design review.","Track reference-heavy directories with snapshot diff reports and clean snapshots before compaction."],"tags":["hdfs","concat","snapshot","inode-reference","rename"],"backgroundTag":"hdfs-snapshot-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}