apache/hadoop · error · FileNotFoundException

File {} does not exist.

Error message

File {} does not exist.

What it means

listStatusIterator() and listLocatedStatus() on the FileContext 'hdfs' provider fetch the first batch of directory entries inside the DirListingIterator constructor. dfs.listPaths returns null when the NameNode has no listing for the path (it is not a live directory in the namespace), and the constructor throws FileNotFoundException before a single element can be produced.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/fs/Hdfs.java:255

   * @param <T> the type of the file status
   */
  abstract private class  DirListingIterator<T extends FileStatus>
  implements RemoteIterator<T> {
    private DirectoryListing thisListing;
    private int i;
    final private String src;
    final private boolean needLocation;  // if status

    private DirListingIterator(Path p, boolean needLocation)
      throws IOException {
      this.src = Hdfs.this.getUriPath(p);
      this.needLocation = needLocation;

      // fetch the first batch of entries in the directory
      thisListing = dfs.listPaths(
          src, HdfsFileStatus.EMPTY_NAME, needLocation);
      if (thisListing == null) { // the directory does not exist
        throw new FileNotFoundException("File " + src + " does not exist.");
      }
    }

    @Override
    public boolean hasNext() throws IOException {
      if (thisListing == null) {
        return false;
      }
      if (i>=thisListing.getPartialListing().length
          && thisListing.hasMore()) { 
        // current listing is exhausted & fetch a new listing
        thisListing = dfs.listPaths(src, thisListing.getLastName(),
            needLocation);
        if (thisListing == null) {
          throw new FileNotFoundException("File " + src + " does not exist.");
        }
        i = 0;
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Gate the call: stat the path with getFileStatus(), require isDirectory(), and only then construct the iterator.
2. If the directory is supposed to exist, create it up front with mkdirs() during setup rather than lazily.
3. If concurrent deletion is legitimate, catch FileNotFoundException and treat the listing as empty/skip.
4. Print the exact qualified path in the failure to catch templating and authority bugs.

Example fix

// before
RemoteIterator<FileStatus> it = fc.listStatusIterator(dir);
// FileNotFoundException: File <dir> does not exist.

// after
FileStatus st;
try {
  st = fc.getFileStatus(dir);
} catch (FileNotFoundException e) {
  return Collections.emptyList();
}
if (!st.isDirectory()) {
  throw new IOException(dir + ' is not a directory');
}
RemoteIterator<FileStatus> it = fc.listStatusIterator(dir);

FileStatus st = fc.getFileStatus(dir); // throws FNFE itself if absent
if (!st.isDirectory()) {
  throw new IOException(dir + " is not a directory");
}
RemoteIterator<FileStatus> it = fc.listStatusIterator(dir);

catch (FileNotFoundException e) {
  // directory absent or not listable: log the qualified path, treat as empty or fail fast
}

When it happens

Trigger: Constructing the RemoteIterator via fc.listStatusIterator(p) or fc.listLocatedStatus(p) when p is missing, was deleted before iteration began, or names something that is not a listable directory. Typical in code that lists an output/partition directory that a failed or not-yet-run upstream step never created.

Common situations: Partition scanners (Hive/Spark style discovery) hitting a date partition that was never written; pipeline stages assuming the previous stage's output dir exists; cleanup jobs deleting dirs between an exists() check and the listing; config-templated paths with substitution typos.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• One or more paths do not exist.
• %s does not exist or is not file.
• Source '{srcFile}' does not exist
• {blockURI}
• Block data not found, r={r}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/14d42da577c57df2. Report an issue: GitHub.