{"record":{"id":"b68d51c0052b0837","repo":"apache/hadoop","slug":"a-disk-io-error-occurred","errorCode":null,"errorMessage":"A disk IO error occurred","messagePattern":"A disk IO error occurred","errorType":"exception","errorClass":"DiskFileCorruptException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java","lineNumber":599,"sourceCode":"      readChecksum(buf, checksumOff, checksumDataLen);\n\n      // write in progress that we need to use to get last checksum\n      if (lastDataPacket && lastChunkChecksum != null) {\n        int start = checksumOff + checksumDataLen - checksumSize;\n        byte[] updatedChecksum = lastChunkChecksum.getChecksum();\n        if (updatedChecksum != null) {\n          System.arraycopy(updatedChecksum, 0, buf, start, checksumSize);\n        }\n      }\n    }\n    \n    int dataOff = checksumOff + checksumDataLen;\n    if (!transferTo) { // normal transfer\n      try {\n        ris.readDataFully(buf, dataOff, dataLen);\n      } catch (IOException ioe) {\n        if (ioe.getMessage().startsWith(EIO_ERROR)) {\n          throw new DiskFileCorruptException(\"A disk IO error occurred\", ioe);\n        }\n        throw ioe;\n      }\n\n      if (verifyChecksum) {\n        verifyChecksum(buf, dataOff, dataLen, numChunks, checksumOff);\n      }\n    }\n    \n    try {\n      if (transferTo) {\n        SocketOutputStream sockOut = (SocketOutputStream)out;\n        // First write header and checksums\n        sockOut.write(buf, headerOff, dataOff - headerOff);\n\n        // no need to flush since we know out is not a buffered stream\n        FileChannel fileCh = ((FileInputStream)ris.getDataIn()).getChannel();\n        LongWritable waitTime = new LongWritable();","sourceCodeStart":581,"sourceCodeEnd":617,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java#L581-L617","documentation":"During a normal (non-transferTo) chunk send, ris.readDataFully hit an IOException whose message starts with the EIO marker (Linux EIO, 'Input/output error'). The datanode wraps it in DiskFileCorruptException because EIO on read almost always means a bad disk track or corrupted file at the page level, not a transient hiccup, and the block should be treated as corrupt rather than retried blindly.","triggerScenarios":"Reading the block data file returns EIO from the kernel: failing sector/disk, corrupted file after a storage fault, or a SAN/NFS-backed volume dropping IO. The wrap happens in the !transferTo branch after readDataFully fails.","commonSituations":"Aging or failing HDDs/SSDs producing unreadable sectors; storage layer faults (RAID degraded, NFS stale); corruption after power loss on non-journaled setups.","solutions":["Check disk health immediately: SMART status, dmesg for I/O errors, and the datanode's volume-failure metrics; replace the disk if failing.","Let the block scanner/NN quarantine the corrupt replica and re-replicate from a healthy copy (verify with fsck).","If only this block's file is corrupt, remove the replica files so re-replication restores it.","After replacing hardware, run a full block scan to catch sibling corruption."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"static boolean isDiskFileCorrupt(IOException e) {\n  return e instanceof DiskFileCorruptException;\n}","tryCatchPattern":"try {\n  sender.sendBlock(out, null, client);\n} catch (DiskFileCorruptException e) {\n  reportCorruptReplicaToNN(block);   // NN quarantines and re-replicates\n  sendFromAlternateReplica(block);   // serve the client from a good copy\n}","preventionTips":["Schedule volume scans (dfs.datanode.scan.period.hours) so corruption is found before clients hit EIO.","Monitor dmesg/SMART for media errors; replace disks at the first EIO, not the tenth.","Keep replication factor above 1 so any single corrupt replica is transparently replaceable."],"tags":["hdfs","datanode","read-path","disk-io","corruption"],"backgroundTag":"disk-io-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}