apache/hadoop · critical · DiskErrorException

Error checking directory ${dir}

Error message

Error checking directory ${dir}

What it means

DiskChecker.checkDirWithDiskIo goes beyond permission checks: doDiskIo writes a 1-byte probe file into the directory, fsyncs it, and deletes it, retrying a few times across generated probe names (getFileNameForDiskIoCheck). Any IOException that survives the retries is rethrown wrapped as DiskErrorException('Error checking directory <dir>', cause). This path runs when the configured disk validator performs real I/O (e.g. 'read-write' via yarn.nodemanager.disk-validator).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java:272

   */
  private static void doDiskIo(File dir) throws DiskErrorException {
    try {
      IOException ioe = null;

      for (int i = 0; i < DISK_IO_MAX_ITERATIONS; ++i) {
        final File file = getFileNameForDiskIoCheck(dir, i+1);
        try {
          diskIoCheckWithoutNativeIo(file);
          return;
        } catch (IOException e) {
          // Let's retry a few times before we really give up and
          // declare the disk as bad.
          ioe = e;
        }
      }
      throw ioe;  // Just rethrow the last exception to signal failure.
    } catch(IOException e) {
      throw new DiskErrorException("Error checking directory " + dir, e);
    }
  }

  /**
   * Try to perform some disk IO by writing to the given file
   * without using Native IO.
   *
   * @param file
   * @throws IOException if there was a non-retriable error.
   */
  private static void diskIoCheckWithoutNativeIo(File file)
      throws IOException {
    FileOutputStream fos = null;

    try {
      final FileIoProvider provider = fileIoProvider.get();
      fos = provider.get(file);
      provider.write(fos, new byte[1]);

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the wrapped cause in DiskErrorException — it distinguishes 'No space left on device' from raw I/O errors
2. df -h the directory; free space or extend the volume if ENOSPC
3. Check dmesg/storage logs for hardware I/O errors; fence and replace the disk if it is failing
4. For transient mount problems, remount the filesystem and let the periodic health check re-evaluate

Example fix

# diagnose: read the cause, then check space and kernel errors
sudo -u yarn df -h /data/nm-local
sudo dmesg | tail -50
# clear space if ENOSPC, else fence the disk and drain the node

// pre-flight: real write probe plus space check before the daemon does it
Path probe = dir.resolve(".preflight-" + System.currentTimeMillis());
Files.write(probe, new byte[1]);
try (FileChannel ch = FileChannel.open(probe, StandardOpenOption.WRITE)) { ch.force(true); }
Files.delete(probe);
if (Files.getFileStore(dir).getUsableSpace() < minBytes) throw new IOException("low space");

try {
  DiskChecker.checkDirWithDiskIo(dir);
} catch (DiskErrorException e) {
  // probe I/O failed after internal retries: fence this directory like NodeManager's DirectoryCollection does
  markDirFailed(dir, e.getCause());
}

When it happens

Trigger: ENOSPC — the filesystem holding the directory is full; EIO from a failing disk during write/sync; filesystem-level errors (journal, overlayfs, NFS glitches) during the probe.

Common situations: DataNode/NodeManager disks filling up and failing the I/O health check; failing storage devices surfacing first under write probes; local dirs on flaky NFS mounts (unsupported but common).

Related errors

• Cannot create directory: ${dir}
• Failed to delete ${file}
• Specified path ${path}does not exist
• Not a directory: ${dir}
• Directory is not readable: ${dir}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8a3d47906c6b5bed. Report an issue: GitHub.