apache/hadoop · error · DiskFileCorruptException

A disk IO error occurred

Error message

A disk IO error occurred

What it means

During a normal (non-transferTo) chunk send, ris.readDataFully hit an IOException whose message starts with the EIO marker (Linux EIO, 'Input/output error'). The datanode wraps it in DiskFileCorruptException because EIO on read almost always means a bad disk track or corrupted file at the page level, not a transient hiccup, and the block should be treated as corrupt rather than retried blindly.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:599

      readChecksum(buf, checksumOff, checksumDataLen);

      // write in progress that we need to use to get last checksum
      if (lastDataPacket && lastChunkChecksum != null) {
        int start = checksumOff + checksumDataLen - checksumSize;
        byte[] updatedChecksum = lastChunkChecksum.getChecksum();
        if (updatedChecksum != null) {
          System.arraycopy(updatedChecksum, 0, buf, start, checksumSize);
        }
      }
    }
    
    int dataOff = checksumOff + checksumDataLen;
    if (!transferTo) { // normal transfer
      try {
        ris.readDataFully(buf, dataOff, dataLen);
      } catch (IOException ioe) {
        if (ioe.getMessage().startsWith(EIO_ERROR)) {
          throw new DiskFileCorruptException("A disk IO error occurred", ioe);
        }
        throw ioe;
      }

      if (verifyChecksum) {
        verifyChecksum(buf, dataOff, dataLen, numChunks, checksumOff);
      }
    }
    
    try {
      if (transferTo) {
        SocketOutputStream sockOut = (SocketOutputStream)out;
        // First write header and checksums
        sockOut.write(buf, headerOff, dataOff - headerOff);

        // no need to flush since we know out is not a buffered stream
        FileChannel fileCh = ((FileInputStream)ris.getDataIn()).getChannel();
        LongWritable waitTime = new LongWritable();

View on GitHub (pinned to 2add963021)

Solutions

1. Check disk health immediately: SMART status, dmesg for I/O errors, and the datanode's volume-failure metrics; replace the disk if failing.
2. Let the block scanner/NN quarantine the corrupt replica and re-replicate from a healthy copy (verify with fsck).
3. If only this block's file is corrupt, remove the replica files so re-replication restores it.
4. After replacing hardware, run a full block scan to catch sibling corruption.

static boolean isDiskFileCorrupt(IOException e) {
  return e instanceof DiskFileCorruptException;
}

try {
  sender.sendBlock(out, null, client);
} catch (DiskFileCorruptException e) {
  reportCorruptReplicaToNN(block);   // NN quarantines and re-replicates
  sendFromAlternateReplica(block);   // serve the client from a good copy
}

When it happens

Trigger: Reading the block data file returns EIO from the kernel: failing sector/disk, corrupted file after a storage fault, or a SAN/NFS-backed volume dropping IO. The wrap happens in the !transferTo branch after readDataFully fails.

Common situations: Aging or failing HDDs/SSDs producing unreadable sectors; storage layer faults (RAID degraded, NFS stale); corruption after power loss on non-journaled setups.

Related errors

• Unknown op {} in data stream
• file VERSION is invalid.
• Replica gen stamp < block genstamp, block={block}, replica={
• Replica is not readable, block={block}, replica={replica}
• Replica not found for {block}. The block may have been remov

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b68d51c0052b0837. Report an issue: GitHub.