{"record":{"id":"06cc3f180c430b48","repo":"apache/hadoop","slug":"disk-check-failed","errorCode":null,"errorMessage":"Disk Check failed!","messagePattern":"Disk Check failed!","errorType":"exception","errorClass":"DiskErrorException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java","lineNumber":82,"sourceCode":"      long writeLatency = TimeUnit.MICROSECONDS.convert(\n          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);\n      metric.addWriteFileLatency(writeLatency);\n\n      // read back\n      startTime = System.nanoTime();\n      byte[] outputBytes = Files.readAllBytes(tmpFile);\n      long readLatency = TimeUnit.MICROSECONDS.convert(\n          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);\n      metric.addReadFileLatency(readLatency);\n\n      // validation\n      if (!Arrays.equals(inputBytes, outputBytes)) {\n        metric.diskCheckFailed();\n        throw new DiskErrorException(\"Data in file has been corrupted.\");\n      }\n    } catch (IOException e) {\n      metric.diskCheckFailed();\n      throw new DiskErrorException(\"Disk Check failed!\", e);\n    } finally {\n      // delete the file\n      if (tmpFile != null) {\n        try {\n          Files.delete(tmpFile);\n        } catch (IOException e) {\n          metric.diskCheckFailed();\n          throw new DiskErrorException(\"File deletion failed!\", e);\n        }\n      }\n    }\n  }\n}\n","sourceCodeStart":64,"sourceCodeEnd":96,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java#L64-L96","documentation":"The catch-all in ReadWriteDiskValidator.checkStatus: any java.io.IOException raised during the probe — Files.createTempFile, writing the 16 bytes, or Files.readAllBytes — increments the failure metric and is rethrown as DiskErrorException(\"Disk Check failed!\", e) with the original IOException as the cause. The specific root cause (ENOSPC, EIO, permissions, read-only) is only visible in getCause().","triggerScenarios":"Volume full (ENOSPC) when creating or writing the temp file; volume remounted read-only after an FS error; ownership/permission changes on the directory; inode exhaustion; SELinux/AppArmor denying file creation.","commonSituations":"NodeManager local dirs filling with localization cache; log dirs filling and cascading into the checker; volumes flipped to ro after a transient I/O error; hardened hosts denying temp-file creation.","solutions":["df -h and df -i the volume; clear space/inodes (cleanup of localization cache, log rotation)","cat /proc/mounts to confirm the filesystem is rw; fix the underlying error (see dmesg) and remount","chown the directory to the daemon user and check audit logs for SELinux/AppArmor denials","Read DiskErrorException.getCause() — it carries the exact original IOException and path"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (dir.getUsableSpace() < MIN_FREE_BYTES || !dir.canWrite()) {\n  throw new IOException(\"volume low on space or not writable: \" + dir);\n}","typeGuard":null,"tryCatchPattern":"try { diskValidator.checkStatus(dir); } catch (DiskErrorException e) { Throwable c = e.getCause(); /* c is the original IOException: ENOSPC/EIO/permission */ volume.markFailed(e.getMessage(), c); }","preventionTips":["Monitor free space and inodes on local/log dirs with alerts well before 100%","Alert on any read-only remount of data volumes","Always inspect getCause() on 'Disk Check failed!' to find the true IO error"],"tags":["hadoop","yarn","disk-check","disk-full","io-error"],"backgroundTag":"disk-io-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}