What causes "No more items in iterator" in apache/hadoop?

Gate every next() with hasNext() — the contract is identical to java.util.Iterator

apache/hadoop · error · NoSuchElementException

No more items in iterator

Error message

No more items in iterator

What it means

NoSuchElementException from the batching RemoteIterator inner class over listStatusBatch (FileSystem.java:2336) that backs FileSystem.listStatusIterator(Path) and listFilesIterator(Path, boolean). The iterator serves entries from one batch at a time (fetchMore() requests the next batch with a token); next() first asserts hasNext() and throws this fixed message when the cursor passed the final entry — it will not fetch another batch past the end.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:2336

    }

    @Override
    public boolean hasNext() throws IOException {
      return i < entries.getEntries().length ||
          entries.hasMore();
    }

    private void fetchMore() throws IOException {
      byte[] token = entries.getToken();
      entries = listStatusBatch(path, token);
      i = 0;
    }

    @Override
    @SuppressWarnings("unchecked")
    public T next() throws IOException {
      if (!hasNext()) {
        throw new NoSuchElementException("No more items in iterator");
      }
      if (i == entries.getEntries().length) {
        fetchMore();
      }
      return (T)entries.getEntries()[i++];
    }
  }

  /**
   * Returns a remote iterator so that followup calls are made on demand
   * while consuming the entries. Each FileSystem implementation should
   * override this method and provide a more efficient implementation, if
   * possible.
   *
   * Does not guarantee to return the iterator that traverses statuses
   * of the files in a sorted order.
   *
   * @param p target path

View on GitHub (pinned to 2add963021)

Solutions

Gate every next() with hasNext() — the contract is identical to java.util.Iterator
For concurrent consumers, drain the RemoteIterator on one thread into a BlockingQueue, or create one iterator per thread
Prefer listStatusIterator/listFilesIterator over hand-rolling listStatusBatch token loops

Example fix

// before
RemoteIterator<FileStatus> it = fs.listStatusIterator(dir);
FileStatus first = it.next(); // throws on empty directory

// after
RemoteIterator<FileStatus> it = fs.listStatusIterator(dir);
FileStatus first = it.hasNext() ? it.next() : null;

Defensive patterns

Strategy: validation

Validate before calling

RemoteIterator<FileStatus> it = fs.listStatusIterator(dir);
FileStatus first = it.hasNext() ? it.next() : null; // guarded first element
while (it.hasNext()) {
  process(it.next());
}

Prevention

Gate every next() with hasNext() — the batch iterator throws rather than fetching past the end
Consume a RemoteIterator from one thread; hand entries to workers via a queue
Do not port array-index loops (i < list.length) onto iterators without adding the hasNext() check

When it happens

Trigger: Calling next() on the iterator returned by listStatusIterator(path) or listFilesIterator(path, recursive) after the last entry, or without checking hasNext(); sharing one RemoteIterator across threads so one consumer drains it and another calls next() unsynchronized.

Common situations: Migrating from the listStatus(Path) array API to iterators for very large directories (Hadoop 3.4 batched listing) and keeping old index-style loops; worker pools consuming a single shared iterator without a handoff queue.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7eb24c7bacf99967. Report an issue: GitHub.