apache/hadoop · error · PathNotFoundException

No such file or directory

Error message

No such file or directory

What it means

PathData.lookupStat (PathData.java:182) wraps FileSystem.getFileStatus(): on FileNotFoundException, when the ignoreFNF flag is false it rethrows as PathNotFoundException ('No such file or directory'). The only current caller with ignoreFNF=false is refreshStatus(), so this fires when re-statting a path whose earlier status was already fetched.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java:182

      inferredSchemeFromPath = checkIfSchemeInferredFromPath(pathString);
    }
  }

  // need a static method for the ctor above
  /**
   * Get the FileStatus info
   * @param ignoreFNF if true, stat will be null if the path doesn't exist
   * @return FileStatus for the given path
   * @throws IOException if anything goes wrong
   */
  private static
  FileStatus lookupStat(FileSystem fs, String pathString, boolean ignoreFNF)
  throws IOException {
    FileStatus status = null;
    try {
      status = fs.getFileStatus(new Path(pathString));
    } catch (FileNotFoundException e) {
      if (!ignoreFNF) throw new PathNotFoundException(pathString);
    }
    // TODO: should consider wrapping other exceptions into Path*Exceptions
    return status;
  }
  
  private void setStat(FileStatus stat) {
    this.stat = stat;
    exists = (stat != null);
  }

  /**
   * Updates the paths's file status
   * @return the updated FileStatus
   * @throws IOException if anything goes wrong...
   */
  public FileStatus refreshStatus() throws IOException {
    FileStatus status = null;
    try {

View on GitHub (pinned to 2add963021)

Solutions

1. Re-verify existence before refreshing: 'if (pd.exists)' is stale — call fs.exists(pd.path) immediately before refreshStatus
2. Catch PathNotFoundException and treat the path as gone (skip/continue) in iterative processing loops
3. Avoid concurrent delete-while-processing: coordinate the pipeline so producers finish before consumers stat files

Example fix

// before
item.refreshStatus(); // throws if deleted since enumeration

// after
try {
  item.refreshStatus();
} catch (PathNotFoundException e) {
  continue; // vanished concurrently, skip it
}

// existence gate immediately before refresh
if (!fs.exists(pd.path)) { /* treat as deleted, skip */ }

catch (PathNotFoundException e) { mark item as vanished and continue the batch; never propagate from a refresh loop }

When it happens

Trigger: Calling PathData.refreshStatus() (or a shell command that calls it, e.g. SetReplication's waitForReplication loop) on a path that was deleted after the PathData was created. TOCTOU: file listed by a glob/expansion, then removed by a concurrent writer/job before the refresh.

Common situations: Shell commands processing globs while another process prunes the tree; long-running -setrep -w waits (refreshStatus runs every 10s) where files get deleted mid-wait; tests and tooling that hold PathData objects across mutating operations.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Is not a directory
• Is a directory
• File/Directory {path} does not exist
• File not found: {}, likely due to delayed block removal
• Directory does not exist: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/79086a39c7d778da. Report an issue: GitHub.