apache/hadoop · error · SnapshotException

"Concat: the source file " + src + " is referred by some oth

Error message

"Concat: the source file " + src + " is referred by some other reference in some snapshot."

What it means

The source inode is an INodeReference (created by rename inside/between snapshot-captured directories) with a reference count above one, so other snapshot views still point at it. verifySrcFiles detects srcINode.isReference() with getReferenceCount() > 1 and throws SnapshotException, because consuming such a file via concat would mutate storage still visible through those other references.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:144

        fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete
      }
      final INode srcINode = iip.getLastINode();
      final INodeFile srcINodeFile = INodeFile.valueOf(srcINode, src);
      // make sure the src file and the target file are in the same dir
      if (srcINodeFile.getParent() != targetParent) {
        throw new HadoopIllegalArgumentException("Source file " + src
            + " is not in the same directory with the target "
            + targetIIP.getPath());
      }
      // make sure all the source files are not in snapshot
      if (srcINode.isInLatestSnapshot(iip.getLatestSnapshotId())) {
        throw new SnapshotException("Concat: the source file " + src
            + " is in snapshot");
      }
      // check if the file has other references.
      if (srcINode.isReference() && ((INodeReference.WithCount)
          srcINode.asReference().getReferredINode()).getReferenceCount() > 1) {
        throw new SnapshotException("Concat: the source file " + src
            + " is referred by some other reference in some snapshot.");
      }
      // source file cannot be the same with the target file
      if (srcINode.equals(targetINode)) {
        throw new HadoopIllegalArgumentException("concat: the src file " + src
            + " is the same with the target file " + targetIIP.getPath());
      }
      // source file cannot be under construction or empty
      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " is invalid or empty or underConstruction");
      }

      // source file's preferred block size cannot be greater than the target
      // file
      if (srcINodeFile.getPreferredBlockSize() >
          targetINode.getPreferredBlockSize()) {
        throw new HadoopIllegalArgumentException("concat: source file " + src

View on GitHub (pinned to 2add963021)

Solutions

Delete the snapshots that hold references to the source (check hdfs lsSnapshotResults / diff reports to find which snapshot references it), dropping the reference count to 1, then concat.
Skip referenced sources in this compaction round and report them for manual handling.
Avoid renaming files within snapshot-captured directories when later concat is planned.

Example fix

// before
fs.rename(new Path("/data/.staging/f"), new Path("/data/f")); // /data has snapshots
fs.concat(target, new Path[]{new Path("/data/f")}); // refcount > 1 -> SnapshotException

// after
// remove snapshots of /data that reference f, then:
fs.concat(target, new Path[]{new Path("/data/f")});

Defensive patterns

Strategy: try-catch

Validate before calling

// references only exist for files renamed under snapshots; avoid the pattern by writing directly
// to the final path instead of temp-then-rename inside snapshot-captured directories.

Try / catch

catch (SnapshotException e) {
  if (e.getMessage() != null && e.getMessage().contains("referred by some other reference")) {
    // this src is shared with a snapshot view: skip it and surface for manual snapshot cleanup
    skipAndReport(src);
  } else { throw e; }
}

Prevention

Do not rename files within directories covered by snapshots when the file may later be a concat source.
Detect rename-under-snapshot patterns in pipelines (temp dir -> final dir where final dir has snapshots) during design review.
Track reference-heavy directories with snapshot diff reports and clean snapshots before compaction.

When it happens

Trigger: Concat on a source that was renamed while a snapshot covered the original location — the rename creates a reference pair (WithCount), leaving the file reachable from both the snapshot path and the current path.

Common situations: Data pipelines that rename files into a final directory that also carries snapshots (classic 'write to temp, rename to final' under snapshot); snapshot-based audit systems on directories that compaction later tries to merge.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/31130f0bfdf336ce. Report an issue: GitHub.