{"record":{"id":"8a3d47906c6b5bed","repo":"apache/hadoop","slug":"error-checking-directory-dir","errorCode":null,"errorMessage":"Error checking directory ${dir}","messagePattern":"Error checking directory (.+?)","errorType":"exception","errorClass":"DiskErrorException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java","lineNumber":272,"sourceCode":"   */\n  private static void doDiskIo(File dir) throws DiskErrorException {\n    try {\n      IOException ioe = null;\n\n      for (int i = 0; i < DISK_IO_MAX_ITERATIONS; ++i) {\n        final File file = getFileNameForDiskIoCheck(dir, i+1);\n        try {\n          diskIoCheckWithoutNativeIo(file);\n          return;\n        } catch (IOException e) {\n          // Let's retry a few times before we really give up and\n          // declare the disk as bad.\n          ioe = e;\n        }\n      }\n      throw ioe;  // Just rethrow the last exception to signal failure.\n    } catch(IOException e) {\n      throw new DiskErrorException(\"Error checking directory \" + dir, e);\n    }\n  }\n\n  /**\n   * Try to perform some disk IO by writing to the given file\n   * without using Native IO.\n   *\n   * @param file\n   * @throws IOException if there was a non-retriable error.\n   */\n  private static void diskIoCheckWithoutNativeIo(File file)\n      throws IOException {\n    FileOutputStream fos = null;\n\n    try {\n      final FileIoProvider provider = fileIoProvider.get();\n      fos = provider.get(file);\n      provider.write(fos, new byte[1]);","sourceCodeStart":254,"sourceCodeEnd":290,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java#L254-L290","documentation":"DiskChecker.checkDirWithDiskIo goes beyond permission checks: doDiskIo writes a 1-byte probe file into the directory, fsyncs it, and deletes it, retrying a few times across generated probe names (getFileNameForDiskIoCheck). Any IOException that survives the retries is rethrown wrapped as DiskErrorException('Error checking directory <dir>', cause). This path runs when the configured disk validator performs real I/O (e.g. 'read-write' via yarn.nodemanager.disk-validator).","triggerScenarios":"ENOSPC — the filesystem holding the directory is full; EIO from a failing disk during write/sync; filesystem-level errors (journal, overlayfs, NFS glitches) during the probe.","commonSituations":"DataNode/NodeManager disks filling up and failing the I/O health check; failing storage devices surfacing first under write probes; local dirs on flaky NFS mounts (unsupported but common).","solutions":["Inspect the wrapped cause in DiskErrorException — it distinguishes 'No space left on device' from raw I/O errors","df -h the directory; free space or extend the volume if ENOSPC","Check dmesg/storage logs for hardware I/O errors; fence and replace the disk if it is failing","For transient mount problems, remount the filesystem and let the periodic health check re-evaluate"],"exampleFix":"# diagnose: read the cause, then check space and kernel errors\nsudo -u yarn df -h /data/nm-local\nsudo dmesg | tail -50\n# clear space if ENOSPC, else fence the disk and drain the node","handlingStrategy":"try-catch","validationCode":"// pre-flight: real write probe plus space check before the daemon does it\nPath probe = dir.resolve(\".preflight-\" + System.currentTimeMillis());\nFiles.write(probe, new byte[1]);\ntry (FileChannel ch = FileChannel.open(probe, StandardOpenOption.WRITE)) { ch.force(true); }\nFiles.delete(probe);\nif (Files.getFileStore(dir).getUsableSpace() < minBytes) throw new IOException(\"low space\");","typeGuard":null,"tryCatchPattern":"try {\n  DiskChecker.checkDirWithDiskIo(dir);\n} catch (DiskErrorException e) {\n  // probe I/O failed after internal retries: fence this directory like NodeManager's DirectoryCollection does\n  markDirFailed(dir, e.getCause());\n}","preventionTips":["Alert on filesystem usage well below 100% so probes never hit ENOSPC","Monitor smartd/dmesg for device errors on nodes running disk I/O validators","Keep Hadoop local dirs on local POSIX filesystems, not NFS"],"tags":["disk","health-check","datanode","nodemanager","io-error","diskchecker"],"backgroundTag":"disk-io-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}