apache/hadoop · error · RuntimeException

Can't check existence of ${next}

Error message

Can't check existence of ${next}

What it means

Thrown by the lazy iterator returned from LocalDirAllocator.getAllLocalPathsToRead(). PathIterator.next() calls fs.exists() on each candidate path under the configured local directories; because Iterator.next() cannot throw checked exceptions, any IOException from that existence check (permissions, vanished mount, disk failure) is wrapped in this RuntimeException.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java:658

      }

      private void advance() throws IOException {
        while (i < rootDirs.length) {
          next = new Path(rootDirs[i++], pathStr);
          if (fs.exists(next)) {
            return;
          }
        }
        next = null;
      }

      @Override
      public Path next() {
        final Path result = next;
        try {
          advance();
        } catch (IOException ie) {
          throw new RuntimeException("Can't check existence of " + next, ie);
        }
        if (result == null) {
          throw new NoSuchElementException();
        }
        return result;
      }

      @Override
      public void remove() {
        throw new UnsupportedOperationException("read only iterator");
      }

      @Override
      public Iterator<Path> iterator() {
        return this;
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

  1. Unwrap the RuntimeException's IOException cause and fix the local directory named in the message (permissions, mount, health)
  2. Verify every dir listed in the configured key exists and is readable by the process user before running the job
  3. Repair/free the failing disk or remount the volume, then rerun
  4. Wrap iteration in try/catch, log the failing dir, and degrade gracefully instead of failing the whole job

Example fix

// before
for (Path p : alloc.getAllLocalPathsToRead(pathStr, conf)) { process(p); }
// after
try {
  for (Path p : alloc.getAllLocalPathsToRead(pathStr, conf)) { process(p); }
} catch (RuntimeException e) {
  if (e.getCause() instanceof IOException) {
    LOG.warn("local dir existence check failed: " + e.getCause());
  } else { throw e; }
}
Defensive patterns

Strategy: try-catch

Validate before calling

for (String d : conf.getTrimmedStrings("yarn.nodemanager.local-dirs")) {
  File f = new File(d);
  if (!f.isDirectory() || !f.canRead()) {
    throw new IOException("local dir unusable: " + d);
  }
}

Try / catch

try {
  for (Path p : alloc.getAllLocalPathsToRead(pathStr, conf)) { /* ... */ }
} catch (RuntimeException e) {
  if (e.getCause() instanceof IOException) {
    LOG.warn("local dir existence check failed: {}", e.getCause());
  } else { throw e; }
}

Prevention

When it happens

Trigger: Iterating the Iterable<Path> from LocalDirAllocator#getAllLocalPathsToRead(String, Configuration) while LocalFileSystem.exists() throws on one of the configured local dirs (context key such as mapreduce.cluster.local.dir / yarn.nodemanager.local-dirs): unreadable directory, unmounted disk, permission change, or I/O error on the volume.

Common situations: Node-local disks failing or full, cleanup scripts deleting configured dir roots, ownership/permission changes after process start, NFS-backed local dirs hiccuping mid-iteration.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/55f93598bf202b9d. Report an issue: GitHub.