apache/hadoop · critical · DiskErrorException

Data in file has been corrupted.

Error message

Data in file has been corrupted.

What it means

The core integrity probe of ReadWriteDiskValidator: it writes 16 random bytes to a fresh temp file in the target directory, reads them back, and compares with Arrays.equals. A mismatch throws DiskErrorException("Data in file has been corrupted.") — the storage returned bytes different from what was written, a strong signal of failing hardware or a broken filesystem layer rather than a configuration problem.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java:78

      byte[] inputBytes = new byte[16];
      RANDOM.nextBytes(inputBytes);
      long startTime = System.nanoTime();
      Files.write(tmpFile, inputBytes);
      long writeLatency = TimeUnit.MICROSECONDS.convert(
          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
      metric.addWriteFileLatency(writeLatency);

      // read back
      startTime = System.nanoTime();
      byte[] outputBytes = Files.readAllBytes(tmpFile);
      long readLatency = TimeUnit.MICROSECONDS.convert(
          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
      metric.addReadFileLatency(readLatency);

      // validation
      if (!Arrays.equals(inputBytes, outputBytes)) {
        metric.diskCheckFailed();
        throw new DiskErrorException("Data in file has been corrupted.");
      }
    } catch (IOException e) {
      metric.diskCheckFailed();
      throw new DiskErrorException("Disk Check failed!", e);
    } finally {
      // delete the file
      if (tmpFile != null) {
        try {
          Files.delete(tmpFile);
        } catch (IOException e) {
          metric.diskCheckFailed();
          throw new DiskErrorException("File deletion failed!", e);
        }
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect dmesg/syslog for I/O errors on that device, then run smartctl -a and fsck
2. Drain the node and replace the failing disk, or drop the directory from local-dirs until repaired
3. If the volume is network-attached, reproduce locally to rule out the mount layer
4. After replacement, confirm ReadWriteDiskValidatorMetrics failure counters stop rising

try { diskValidator.checkStatus(dir); } catch (DiskErrorException e) { volume.markFailed(); ops.alert("possible disk corruption on " + dir + ": " + e.getMessage()); }

When it happens

Trigger: checkStatus on a volume whose reads do not return what was written: dying disk, faulty cable/controller/NVMe, buggy NFS/CIFS caching layers, severe memory errors corrupting data before it reaches the platter.

Common situations: NodeManager local volumes degrading on aging worker nodes; cheap drives in dense clusters; network filesystems with coherence bugs; the error recurs every disk-check interval until the volume is removed or repaired.

Related errors

• dir + " is not a directory!"
• Disk Check failed!
• File deletion failed!
• Checksum file not a length multiple of checksum size in {} a
• Checksum error: {} at {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3a8160d85483de26. Report an issue: GitHub.