{"record":{"id":"b450cbb2aed9058e","repo":"apache/hadoop","slug":"one-or-more-paths-do-not-exist","errorCode":null,"errorMessage":"One or more paths do not exist.","messagePattern":"One or more paths do not exist\\.","errorType":"exception","errorClass":"FileNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java","lineNumber":1502,"sourceCode":"    private int listingIdx = 0;\n\n    PartialListingIterator(List<Path> paths, boolean needLocation)\n        throws IOException {\n      this.paths = paths;\n      this.srcs = new String[paths.size()];\n      for (int i = 0; i < paths.size(); i++) {\n        this.srcs[i] = getPathName(paths.get(i));\n      }\n      this.needLocation = needLocation;\n\n      // Do the first listing\n      statistics.incrementReadOps(1);\n      storageStatistics.incrementOpCounter(OpType.LIST_LOCATED_STATUS);\n      batchedListing = dfs.batchedListPaths(\n          srcs, HdfsFileStatus.EMPTY_NAME, needLocation);\n      LBI_LOG.trace(\"Got batchedListing: {}\", batchedListing);\n      if (batchedListing == null) { // the directory does not exist\n        throw new FileNotFoundException(\"One or more paths do not exist.\");\n      }\n    }\n\n    @Override\n    public boolean hasNext() throws IOException {\n      if (batchedListing == null) {\n        return false;\n      }\n      // If we're done with the current batch, try to get the next batch\n      if (listingIdx >= batchedListing.getListings().length) {\n        if (!batchedListing.hasMore()) {\n          LBI_LOG.trace(\"No more elements\");\n          return false;\n        }\n        batchedListing = dfs.batchedListPaths(\n            srcs, batchedListing.getStartAfter(), needLocation);\n        LBI_LOG.trace(\"Got batchedListing: {}\", batchedListing);\n        listingIdx = 0;","sourceCodeStart":1484,"sourceCodeEnd":1520,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L1484-L1520","documentation":"PartialListingIterator (batchedListStatusIterator/batchedListLocatedStatusIterator) sends all requested paths in one getBatchedListing RPC; the NameNode returns null when any of the paths does not exist, so the constructor throws FileNotFoundException naming 'one or more paths'. The batched API is all-or-nothing at startup: one bad path fails the whole call even if later batches would have carried per-path exceptions.","triggerScenarios":"Calling batchedListStatusIterator(List<Path>) or batchedListLocatedStatusIterator(List<Path>) where at least one path in the list is missing, deleted, mistyped, or is a file rather than a directory.","commonSituations":"Partition-pruning code that builds path lists from stale catalog metadata (Hive/Spark partitions dropped); enumerating many user directories where one was removed between discovery and listing; mixing qualified and unqualified paths against the wrong fs.defaultFS.","solutions":["Pre-validate each path with exists() (or one getFileStatus sweep) and drop the missing ones before calling the batched iterator.","Re-fetch/refresh the path list from the source of truth if it may be stale.","Catch FileNotFoundException, fall back to listing the surviving paths, or fall back to per-path listStatusIterator calls.","Fix wrong paths caused by fs.defaultFS or missing scheme/authority."],"exampleFix":"// before\nRemoteIterator<PartialListing<FileStatus>> it = fs.batchedListStatusIterator(paths);\n\n// after\nList<Path> valid = new ArrayList<>();\nfor (Path p : paths) {\n  if (fs.exists(p)) valid.add(p);\n}\nif (valid.isEmpty()) return Collections.emptyList();\nRemoteIterator<PartialListing<FileStatus>> it = fs.batchedListStatusIterator(valid);","handlingStrategy":"validation","validationCode":"List<Path> valid = new ArrayList<>();\nfor (Path p : paths) {\n  if (fs.exists(p)) valid.add(p);\n}\nif (valid.isEmpty()) return Collections.emptyList();\n// call batched iterator with 'valid' only","typeGuard":null,"tryCatchPattern":"try {\n  it = fs.batchedListStatusIterator(paths);\n} catch (FileNotFoundException e) {\n  // one or more paths missing: fall back to filtered per-path iteration\n}","preventionTips":["Batched listing is all-or-nothing at construction: pre-validate every path when the list is stale.","Refresh path lists from the source of truth (catalog/metastore) immediately before listing.","Watch for per-path errors delivered as PartialListing exceptions later — absence is the only startup-fatal case."],"tags":["hdfs","batched-listing","file-not-found","remote-iterator","distributed-filesystem"],"backgroundTag":"file-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}