What causes "File does not exist: {} (inode {}) {}" in apache/hadoop?

Abort this stream and restart the output under a fresh path — the target is gone

apache/hadoop · error · FileNotFoundException

File does not exist: {} (inode {}) {}

Error message

File does not exist: {} (inode {}) {}

What it means

checkLease — the guard for addBlock/abandonBlock/close/recoverLease on a held lease — resolved the path but iip.getLastINode() is null: the file was deleted from the namespace while the client still held its lease. FileNotFoundException('File does not exist: ...') includes the inode id and the holder's lease state for diagnosis.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:3253

    } finally {
      writeUnlock(RwLockMode.GLOBAL, operationName);
    }
    getEditLog().logSync();
  }

  private String leaseExceptionString(String src, long fileId, String holder) {
    final Lease lease = leaseManager.getLease(holder);
    return src + " (inode " + fileId + ") " + (lease != null? lease.toString()
        : "Holder " + holder + " does not have any open files.");
  }

  INodeFile checkLease(INodesInPath iip, String holder, long fileId)
      throws LeaseExpiredException, FileNotFoundException {
    String src = iip.getPath();
    INode inode = iip.getLastINode();
    assert hasReadLock(RwLockMode.FS);
    if (inode == null) {
      throw new FileNotFoundException("File does not exist: "
          + leaseExceptionString(src, fileId, holder));
    }
    if (!inode.isFile()) {
      throw new LeaseExpiredException("INode is not a regular file: "
          + leaseExceptionString(src, fileId, holder));
    }
    final INodeFile file = inode.asFile();
    if (!file.isUnderConstruction()) {
      throw new LeaseExpiredException("File is not open for writing: "
          + leaseExceptionString(src, fileId, holder));
    }
    // No further modification is allowed on a deleted file.
    // A file is considered deleted, if it is not in the inodeMap or is marked
    // as deleted in the snapshot feature.
    if (isFileDeleted(file)) {
      throw new FileNotFoundException("File is deleted: "
          + leaseExceptionString(src, fileId, holder));
    }

View on GitHub (pinned to 2add963021)

Solutions

Abort this stream and restart the output under a fresh path — the target is gone
Coordinate lifecycle: writers own their temp paths; cleaners only delete paths that are closed (check DistributedFileSystem#isFileClosed)
Find who deleted it: the NameNode audit log records the delete RPC with user, path, and time

Example fix

// before
out.close();   // or out.write -> FileNotFoundException mid-stream
// after
try { out.close(); }
catch (FileNotFoundException e) {
  LOG.error("output deleted mid-write: {}", path);
  restartWriteUnder(newPath);
}

Defensive patterns

Strategy: try-catch

Try / catch

catch (FileNotFoundException e) { // target deleted mid-write: abort this stream, re-create under a new name, rewrite from last committed offset }

Prevention

Use the own-and-rename pattern: write to a private temp path, rename on success
Never point cleanup jobs at paths that may be under active lease (check isFileClosed first)

When it happens

Trigger: A writer streams to a file that another actor deletes mid-write (user rm, cleanup job, trash purge, rename-overwrite); the writer's next addBlock/close/recoverLease then hits the missing inode.

Common situations: Concurrent cleanup jobs deleting temp/part files still being written; a user deletes a file an ETL is streaming into; a directory is deleted after a job was preempted but before its close.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2d4de75c9b4a9462. Report an issue: GitHub.