apache/hadoop · error · SnapshotException

"Concat: the source file " + src + " is in snapshot"

Error message

"Concat: the source file " + src + " is in snapshot"

What it means

A source file that appears in the latest snapshot cannot be concat-consumed: verifySrcFiles calls srcINode.isInLatestSnapshot(iip.getLatestSnapshotId()) and throws SnapshotException. Concat effectively deletes each source inode after splicing its blocks; deleting an inode that a snapshot preserves would corrupt the snapshot's view of the directory. The NameNode therefore protects snapshot-captured files.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:138

    // now check the srcs
    for(String src : srcs) {
      final INodesInPath iip = fsd.resolvePath(pc, src, DirOp.WRITE);
      // permission check for srcs
      if (pc != null && fsd.isPermissionEnabled()) {
        fsd.checkPathAccess(pc, iip, FsAction.READ); // read the file
        fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete
      }
      final INode srcINode = iip.getLastINode();
      final INodeFile srcINodeFile = INodeFile.valueOf(srcINode, src);
      // make sure the src file and the target file are in the same dir
      if (srcINodeFile.getParent() != targetParent) {
        throw new HadoopIllegalArgumentException("Source file " + src
            + " is not in the same directory with the target "
            + targetIIP.getPath());
      }
      // make sure all the source files are not in snapshot
      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {
        throw new SnapshotException("Concat: the source file " + src
            + " is in snapshot");
      }
      // check if the file has other references.
      if (srcINode.isReference() && ((INodeReference.WithCount)
          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {
        throw new SnapshotException("Concat: the source file " + src
            + " is referred by some other reference in some snapshot.");
      }
      // source file cannot be the same with the target file
      if (srcINode.equals(targetINode)) {
        throw new HadoopIllegalArgumentException("concat: the src file " + src
            + " is the same with the target file " + targetIIP.getPath());
      }
      // source file cannot be under construction or empty
      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " is invalid or empty or underConstruction");
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the snapshots covering the source files first (hdfs dfs -deleteSnapshot <dir> <snapshotName>), then re-run concat.
2. Pick source files created after the newest snapshot in that directory.
3. Keep snapshot schedules and compaction windows disjoint: compact only files known to post-date every retained snapshot.

Example fix

# before
hdfs dfs -createSnapshot /data snap1
hadoop jar compactor.jar concat /data/part-0000 /data/part-0001  # part-0001 in snap1 -> SnapshotException

# after
hdfs dfs -deleteSnapshot /data snap1
hadoop jar compactor.jar concat /data/part-0000 /data/part-0001

// no direct client API reports 'file is in latest snapshot'; plan around snapshot windows instead:
// compact only files whose mtime is newer than the newest snapshot of the directory
FileStatus newestSnap = newestSnapshotOf(dir); // from SnapshotDescription listing
if (st.getModificationTime() > newestSnapTime) { candidates.add(p); }

catch (SnapshotException e) {
  if (e.getMessage() != null && e.getMessage().contains("is in snapshot")) {
    // exclude this src and continue with the rest; requires NameNode-side retry of the batch without it
    skipAndReport(src);
  } else { throw e; }
}

When it happens

Trigger: FileSystem.concat where a src file (or an ancestor captured in a snapshot) was snapshotted at or after its creation: any snapshot of the parent directory that includes the src triggers the check.

Common situations: Backup/point-in-time snapshots enabled on data directories while a compaction job also runs concat over those directories; demo/dev clusters with frequent snapshots where compaction suddenly fails after the first snapshot is taken.

Related errors

• "Concat: the source file " + src + " is referred by some oth
• {} doesn't support createSnapshot
• {} doesn't support renameSnapshot
• {} doesn't support deleteSnapshot
• Target path not specified. <target path> <src path> <src pat

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/790e6ce3c1a9dbc6. Report an issue: GitHub.