apache/hadoop · error · FileNotFoundException

File not found: {}, likely due to delayed block removal

Error message

File not found: {}, likely due to delayed block removal

What it means

In commitBlockSynchronization, the block's inode resolves but isFileDeleted reports the file's path was removed (or snapshot-deleted) while its blocks await delayed removal. FileNotFoundException is thrown so no CloseOp edit is logged for an already-deleted file (HDFS-6825 family).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:4113

      // first removes the file paths from namespace, and delays the removal
      // of blocks to later time for better performance. When
      // commitBlockSynchronization (this method) is called in between, the
      // blockCollection of storedBlock could have been assigned to null by
      // the delete operation, throw IOException here instead of NPE; if the
      // file path is already removed from namespace by the delete operation,
      // throw FileNotFoundException here, so not to proceed to the end of
      // this method to add a CloseOp to the edit log for an already deleted
      // file (See HDFS-6825).
      //
      if (storedBlock.isDeleted()) {
        throw new IOException("The blockCollection of " + storedBlock
            + " is null, likely because the file owning this block was"
            + " deleted and the block removal is delayed");
      }
      final INodeFile iFile = getBlockCollection(storedBlock);
      src = iFile.getFullPathName();
      if (isFileDeleted(iFile)) {
        throw new FileNotFoundException("File not found: "
            + src + ", likely due to delayed block removal");
      }
      if ((!iFile.isUnderConstruction() || storedBlock.isComplete()) &&
          iFile.getLastBlock().isComplete()) {
        if (LOG.isDebugEnabled()) {
          LOG.debug("Unexpected block (={}) since the file (={}) is not under construction",
              oldBlock, iFile.getLocalName());
        }

        return;
      }

      truncatedBlock = iFile.getLastBlock();
      final long recoveryId = truncatedBlock.getUnderConstructionFeature()
          .getBlockRecoveryId();
      copyTruncate = truncatedBlock.getBlockId() != storedBlock.getBlockId();
      if(recoveryId != newgenerationstamp) {
        throw new IOException("The recovery id " + newgenerationstamp

View on GitHub (pinned to 2add963021)

Solutions

  1. Confirm with getFileStatus whether the path is really gone; if yes, abandon the write
  2. Retry the client operation only after confirming the file survives and the delete storm is over
  3. Move deletes and finalize/close into the same serialized section of the application
Defensive patterns

Strategy: validation

Validate before calling

for (String p : srcs) {
  if (!fs.exists(new Path(p))) { /* skip or fail early with clear cause */ }
}

Try / catch

try {
  commitBlockSynchronization(oldBlock, newGs, newLen, ...);
} catch (FileNotFoundException e) {
  if (e.getMessage().contains("delayed block removal")) { return; }
  throw e;
}

Prevention

When it happens

Trigger: Same delete-vs-commit race as the sibling checks, but reached one step later: the block still has its inode attached while the path is already gone.

Common situations: File deleted concurrently with block-synchronization retries; snapshot deletions pinning the inode after the path is removed.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b1e8ee9e3f5c4cb8. Report an issue: GitHub.