apache/hadoop · error · FileNotFoundException

One or more paths do not exist.

Error message

One or more paths do not exist.

What it means

PartialListingIterator (batchedListStatusIterator/batchedListLocatedStatusIterator) sends all requested paths in one getBatchedListing RPC; the NameNode returns null when any of the paths does not exist, so the constructor throws FileNotFoundException naming 'one or more paths'. The batched API is all-or-nothing at startup: one bad path fails the whole call even if later batches would have carried per-path exceptions.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:1502

    private int listingIdx = 0;

    PartialListingIterator(List<Path> paths, boolean needLocation)
        throws IOException {
      this.paths = paths;
      this.srcs = new String[paths.size()];
      for (int i = 0; i < paths.size(); i++) {
        this.srcs[i] = getPathName(paths.get(i));
      }
      this.needLocation = needLocation;

      // Do the first listing
      statistics.incrementReadOps(1);
      storageStatistics.incrementOpCounter(OpType.LIST_LOCATED_STATUS);
      batchedListing = dfs.batchedListPaths(
          srcs, HdfsFileStatus.EMPTY_NAME, needLocation);
      LBI_LOG.trace("Got batchedListing: {}", batchedListing);
      if (batchedListing == null) { // the directory does not exist
        throw new FileNotFoundException("One or more paths do not exist.");
      }
    }

    @Override
    public boolean hasNext() throws IOException {
      if (batchedListing == null) {
        return false;
      }
      // If we're done with the current batch, try to get the next batch
      if (listingIdx >= batchedListing.getListings().length) {
        if (!batchedListing.hasMore()) {
          LBI_LOG.trace("No more elements");
          return false;
        }
        batchedListing = dfs.batchedListPaths(
            srcs, batchedListing.getStartAfter(), needLocation);
        LBI_LOG.trace("Got batchedListing: {}", batchedListing);
        listingIdx = 0;

View on GitHub (pinned to 2add963021)

Solutions

1. Pre-validate each path with exists() (or one getFileStatus sweep) and drop the missing ones before calling the batched iterator.
2. Re-fetch/refresh the path list from the source of truth if it may be stale.
3. Catch FileNotFoundException, fall back to listing the surviving paths, or fall back to per-path listStatusIterator calls.
4. Fix wrong paths caused by fs.defaultFS or missing scheme/authority.

Example fix

// before
RemoteIterator<PartialListing<FileStatus>> it = fs.batchedListStatusIterator(paths);

// after
List<Path> valid = new ArrayList<>();
for (Path p : paths) {
  if (fs.exists(p)) valid.add(p);
}
if (valid.isEmpty()) return Collections.emptyList();
RemoteIterator<PartialListing<FileStatus>> it = fs.batchedListStatusIterator(valid);

List<Path> valid = new ArrayList<>();
for (Path p : paths) {
  if (fs.exists(p)) valid.add(p);
}
if (valid.isEmpty()) return Collections.emptyList();
// call batched iterator with 'valid' only

try {
  it = fs.batchedListStatusIterator(paths);
} catch (FileNotFoundException e) {
  // one or more paths missing: fall back to filtered per-path iteration
}

When it happens

Trigger: Calling batchedListStatusIterator(List<Path>) or batchedListLocatedStatusIterator(List<Path>) where at least one path in the list is missing, deleted, mistyped, or is a file rather than a directory.

Common situations: Partition-pruning code that builds path lists from stale catalog metadata (Hive/Spark partitions dropped); enumerating many user directories where one was removed between discovery and listing; mixing qualified and unqualified paths against the wrong fs.defaultFS.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Path {} does not exist
• File {} does not exist.
• No more entries
• File does not exist:
• %s does not exist or is not file.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b450cbb2aed9058e. Report an issue: GitHub.