{"record":{"id":"790e6ce3c1a9dbc6","repo":"apache/hadoop","slug":"concat-the-source-file-src-is-in-snapsho","errorCode":null,"errorMessage":"\"Concat: the source file \" + src + \" is in snapshot\"","messagePattern":"\"Concat: the source file \" \\+ src \\+ \" is in snapshot\"","errorType":"exception","errorClass":"SnapshotException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":138,"sourceCode":"    // now check the srcs\n    for(String src : srcs) {\n      final INodesInPath iip = fsd.resolvePath(pc, src, DirOp.WRITE);\n      // permission check for srcs\n      if (pc != null && fsd.isPermissionEnabled()) {\n        fsd.checkPathAccess(pc, iip, FsAction.READ); // read the file\n        fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete\n      }\n      final INode srcINode = iip.getLastINode();\n      final INodeFile srcINodeFile = INodeFile.valueOf(srcINode, src);\n      // make sure the src file and the target file are in the same dir\n      if (srcINodeFile.getParent() != targetParent) {\n        throw new HadoopIllegalArgumentException(\"Source file \" + src\n            + \" is not in the same directory with the target \"\n            + targetIIP.getPath());\n      }\n      // make sure all the source files are not in snapshot\n      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is in snapshot\");\n      }\n      // check if the file has other references.\n      if (srcINode.isReference() && ((INodeReference.WithCount)\n          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {\n        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is referred by some other reference in some snapshot.\");\n      }\n      // source file cannot be the same with the target file\n      if (srcINode.equals(targetINode)) {\n        throw new HadoopIllegalArgumentException(\"concat: the src file \" + src\n            + \" is the same with the target file \" + targetIIP.getPath());\n      }\n      // source file cannot be under construction or empty\n      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" is invalid or empty or underConstruction\");\n      }","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L120-L156","documentation":"A source file that appears in the latest snapshot cannot be concat-consumed: verifySrcFiles calls srcINode.isInLatestSnapshot(iip.getLatestSnapshotId()) and throws SnapshotException. Concat effectively deletes each source inode after splicing its blocks; deleting an inode that a snapshot preserves would corrupt the snapshot's view of the directory. The NameNode therefore protects snapshot-captured files.","triggerScenarios":"FileSystem.concat where a src file (or an ancestor captured in a snapshot) was snapshotted at or after its creation: any snapshot of the parent directory that includes the src triggers the check.","commonSituations":"Backup/point-in-time snapshots enabled on data directories while a compaction job also runs concat over those directories; demo/dev clusters with frequent snapshots where compaction suddenly fails after the first snapshot is taken.","solutions":["Delete the snapshots covering the source files first (hdfs dfs -deleteSnapshot <dir> <snapshotName>), then re-run concat.","Pick source files created after the newest snapshot in that directory.","Keep snapshot schedules and compaction windows disjoint: compact only files known to post-date every retained snapshot."],"exampleFix":"# before\nhdfs dfs -createSnapshot /data snap1\nhadoop jar compactor.jar concat /data/part-0000 /data/part-0001  # part-0001 in snap1 -> SnapshotException\n\n# after\nhdfs dfs -deleteSnapshot /data snap1\nhadoop jar compactor.jar concat /data/part-0000 /data/part-0001","handlingStrategy":"try-catch","validationCode":"// no direct client API reports 'file is in latest snapshot'; plan around snapshot windows instead:\n// compact only files whose mtime is newer than the newest snapshot of the directory\nFileStatus newestSnap = newestSnapshotOf(dir); // from SnapshotDescription listing\nif (st.getModificationTime() > newestSnapTime) { candidates.add(p); }","typeGuard":null,"tryCatchPattern":"catch (SnapshotException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"is in snapshot\")) {\n    // exclude this src and continue with the rest; requires NameNode-side retry of the batch without it\n    skipAndReport(src);\n  } else { throw e; }\n}","preventionTips":["Keep snapshot schedules and concat/compaction windows disjoint; compact files created after the latest snapshot.","Before compaction, list snapshots of the directory (fs.getSnapshotListing equivalent / hdfs lsSnapshot) and understand coverage.","Delete obsolete snapshots before large compaction rounds so inode cleanup is not blocked."],"tags":["hdfs","concat","snapshot","snapshot-conflict"],"backgroundTag":"hdfs-snapshot-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}