{"record":{"id":"14d42da577c57df2","repo":"apache/hadoop","slug":"file-does-not-exist-14d42d","errorCode":null,"errorMessage":"File {} does not exist.","messagePattern":"File (.+?) does not exist\\.","errorType":"exception","errorClass":"FileNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/fs/Hdfs.java","lineNumber":255,"sourceCode":"   * @param <T> the type of the file status\n   */\n  abstract private class  DirListingIterator<T extends FileStatus>\n  implements RemoteIterator<T> {\n    private DirectoryListing thisListing;\n    private int i;\n    final private String src;\n    final private boolean needLocation;  // if status\n\n    private DirListingIterator(Path p, boolean needLocation)\n      throws IOException {\n      this.src = Hdfs.this.getUriPath(p);\n      this.needLocation = needLocation;\n\n      // fetch the first batch of entries in the directory\n      thisListing = dfs.listPaths(\n          src, HdfsFileStatus.EMPTY_NAME, needLocation);\n      if (thisListing == null) { // the directory does not exist\n        throw new FileNotFoundException(\"File \" + src + \" does not exist.\");\n      }\n    }\n\n    @Override\n    public boolean hasNext() throws IOException {\n      if (thisListing == null) {\n        return false;\n      }\n      if (i>=thisListing.getPartialListing().length\n          && thisListing.hasMore()) { \n        // current listing is exhausted & fetch a new listing\n        thisListing = dfs.listPaths(src, thisListing.getLastName(),\n            needLocation);\n        if (thisListing == null) {\n          throw new FileNotFoundException(\"File \" + src + \" does not exist.\");\n        }\n        i = 0;\n      }","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/fs/Hdfs.java#L237-L273","documentation":"listStatusIterator() and listLocatedStatus() on the FileContext 'hdfs' provider fetch the first batch of directory entries inside the DirListingIterator constructor. dfs.listPaths returns null when the NameNode has no listing for the path (it is not a live directory in the namespace), and the constructor throws FileNotFoundException before a single element can be produced.","triggerScenarios":"Constructing the RemoteIterator via fc.listStatusIterator(p) or fc.listLocatedStatus(p) when p is missing, was deleted before iteration began, or names something that is not a listable directory. Typical in code that lists an output/partition directory that a failed or not-yet-run upstream step never created.","commonSituations":"Partition scanners (Hive/Spark style discovery) hitting a date partition that was never written; pipeline stages assuming the previous stage's output dir exists; cleanup jobs deleting dirs between an exists() check and the listing; config-templated paths with substitution typos.","solutions":["Gate the call: stat the path with getFileStatus(), require isDirectory(), and only then construct the iterator.","If the directory is supposed to exist, create it up front with mkdirs() during setup rather than lazily.","If concurrent deletion is legitimate, catch FileNotFoundException and treat the listing as empty/skip.","Print the exact qualified path in the failure to catch templating and authority bugs."],"exampleFix":"// before\nRemoteIterator<FileStatus> it = fc.listStatusIterator(dir);\n// FileNotFoundException: File <dir> does not exist.\n\n// after\nFileStatus st;\ntry {\n  st = fc.getFileStatus(dir);\n} catch (FileNotFoundException e) {\n  return Collections.emptyList();\n}\nif (!st.isDirectory()) {\n  throw new IOException(dir + ' is not a directory');\n}\nRemoteIterator<FileStatus> it = fc.listStatusIterator(dir);","handlingStrategy":"validation","validationCode":"FileStatus st = fc.getFileStatus(dir); // throws FNFE itself if absent\nif (!st.isDirectory()) {\n  throw new IOException(dir + \" is not a directory\");\n}\nRemoteIterator<FileStatus> it = fc.listStatusIterator(dir);","typeGuard":null,"tryCatchPattern":"catch (FileNotFoundException e) {\n  // directory absent or not listable: log the qualified path, treat as empty or fail fast\n}","preventionTips":["Gate every listing on getFileStatus().isDirectory().","Create output/partition directories during producer setup, not lazily at consumer time.","In pipelines, make 'directory exists' part of the producer's completion contract."],"tags":["hdfs","directory-listing","file-not-found","remote-iterator","filecontext"],"backgroundTag":"file-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}