What causes "File does not exist: " in apache/hadoop?

Call fs.exists(p) (or catch FileNotFoundException) when absence is a normal case and skip/log.

apache/hadoop · error · FileNotFoundException

File does not exist:

Error message

File does not exist:

What it means

getFileStatus(Path) calls dfs.getFileInfo; the NameNode returns null when the path does not exist and the client throws FileNotFoundException with the offending path. This is the canonical HDFS 'stat failed' error and is the exception fs.exists() is built on top of.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:1932

   * and it will get the resolved path's FileStatus object. It will not be
   * represented as a symlink and isDirectory API returns true if the resolved
   * path is a directory, false otherwise.
   *
   * @throws FileNotFoundException if the file does not exist.
   */
  @Override
  public FileStatus getFileStatus(Path f) throws IOException {
    statistics.incrementReadOps(1);
    storageStatistics.incrementOpCounter(OpType.GET_FILE_STATUS);
    Path absF = fixRelativePart(f);
    return new FileSystemLinkResolver<FileStatus>() {
      @Override
      public FileStatus doCall(final Path p) throws IOException {
        HdfsFileStatus fi = dfs.getFileInfo(getPathName(p));
        if (fi != null) {
          return fi.makeQualified(getUri(), p);
        } else {
          throw new FileNotFoundException("File does not exist: " + p);
        }
      }
      @Override
      public FileStatus next(final FileSystem fs, final Path p)
          throws IOException {
        return fs.getFileStatus(p);
      }
    }.resolve(this, absF);
  }

  /**
   * Synchronize client metadata state with Active NameNode.
   * <p>
   * In HA the client synchronizes its state with the Active NameNode
   * in order to guarantee subsequent read consistency from Observer Nodes.
   * @throws IOException
   */
  @Override

View on GitHub (pinned to 2add963021)

Solutions

Call fs.exists(p) (or catch FileNotFoundException) when absence is a normal case and skip/log.
Fix the path construction: fully qualify with fs.makeQualified(p) or set fs.defaultFS correctly.
If racing with deletions, coordinate so readers finish before files are removed, or read from an HDFS snapshot path.
Verify the file was actually created upstream (check job output, _SUCCESS, distcp -update results).

Example fix

// before
FileStatus st = fs.getFileStatus(path);

// after
FileStatus st;
try {
  st = fs.getFileStatus(path);
} catch (FileNotFoundException e) {
  LOG.warn("Missing path, skipping: {}", path);
  return Optional.empty();
}

Defensive patterns

Strategy: validation

Validate before calling

if (fs.exists(p)) {
  FileStatus st = fs.getFileStatus(p);
} else {
  // handle absence before calling
}

Try / catch

try {
  FileStatus st = fs.getFileStatus(p);
} catch (FileNotFoundException e) {
  // normal-case absence: skip, default, or create the file
}

Prevention

Use fs.exists() when absence is expected; use try-catch when the common case is that the file exists (avoids a second RPC).
Qualify paths explicitly (scheme+authority) in multi-cluster code.
Audit jobs that read paths derived from catalogs for races with retention/deletion jobs.

When it happens

Trigger: Calling getFileStatus (directly, or via fs.exists-adjacent code, FileInputFormat splits, Hive/Spark partition metadata checks) on a path absent from the namespace: deleted, never created, typo, or resolved against the wrong fs.defaultFS.

Common situations: Jobs reading partitions that were dropped by a retention job; wrong fs.defaultFS so absolute paths resolve to another cluster; paths with typos or missing scheme/authority; files deleted between a catalog update and the read.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/af47350da94cd649. Report an issue: GitHub.