apache/hadoop · error · DiskErrorException

Disk Check failed!

Error message

Disk Check failed!

What it means

The catch-all in ReadWriteDiskValidator.checkStatus: any java.io.IOException raised during the probe — Files.createTempFile, writing the 16 bytes, or Files.readAllBytes — increments the failure metric and is rethrown as DiskErrorException("Disk Check failed!", e) with the original IOException as the cause. The specific root cause (ENOSPC, EIO, permissions, read-only) is only visible in getCause().

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java:82

      long writeLatency = TimeUnit.MICROSECONDS.convert(
          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
      metric.addWriteFileLatency(writeLatency);

      // read back
      startTime = System.nanoTime();
      byte[] outputBytes = Files.readAllBytes(tmpFile);
      long readLatency = TimeUnit.MICROSECONDS.convert(
          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
      metric.addReadFileLatency(readLatency);

      // validation
      if (!Arrays.equals(inputBytes, outputBytes)) {
        metric.diskCheckFailed();
        throw new DiskErrorException("Data in file has been corrupted.");
      }
    } catch (IOException e) {
      metric.diskCheckFailed();
      throw new DiskErrorException("Disk Check failed!", e);
    } finally {
      // delete the file
      if (tmpFile != null) {
        try {
          Files.delete(tmpFile);
        } catch (IOException e) {
          metric.diskCheckFailed();
          throw new DiskErrorException("File deletion failed!", e);
        }
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. df -h and df -i the volume; clear space/inodes (cleanup of localization cache, log rotation)
2. cat /proc/mounts to confirm the filesystem is rw; fix the underlying error (see dmesg) and remount
3. chown the directory to the daemon user and check audit logs for SELinux/AppArmor denials
4. Read DiskErrorException.getCause() — it carries the exact original IOException and path

if (dir.getUsableSpace() < MIN_FREE_BYTES || !dir.canWrite()) {
  throw new IOException("volume low on space or not writable: " + dir);
}

try { diskValidator.checkStatus(dir); } catch (DiskErrorException e) { Throwable c = e.getCause(); /* c is the original IOException: ENOSPC/EIO/permission */ volume.markFailed(e.getMessage(), c); }

When it happens

Trigger: Volume full (ENOSPC) when creating or writing the temp file; volume remounted read-only after an FS error; ownership/permission changes on the directory; inode exhaustion; SELinux/AppArmor denying file creation.

Common situations: NodeManager local dirs filling with localization cache; log dirs filling and cascading into the checker; volumes flipped to ro after a transient I/O error; hardened hosts denying temp-file creation.

Related errors

• dir + " is not a directory!"
• Data in file has been corrupted.
• File deletion failed!
• Input/output error
• Failed to copy {srcFile} to {destFile} due to failure in Nat

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/06cc3f180c430b48. Report an issue: GitHub.