{"record":{"id":"3a8160d85483de26","repo":"apache/hadoop","slug":"data-in-file-has-been-corrupted","errorCode":null,"errorMessage":"Data in file has been corrupted.","messagePattern":"Data in file has been corrupted\\.","errorType":"exception","errorClass":"DiskErrorException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java","lineNumber":78,"sourceCode":"      byte[] inputBytes = new byte[16];\n      RANDOM.nextBytes(inputBytes);\n      long startTime = System.nanoTime();\n      Files.write(tmpFile, inputBytes);\n      long writeLatency = TimeUnit.MICROSECONDS.convert(\n          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);\n      metric.addWriteFileLatency(writeLatency);\n\n      // read back\n      startTime = System.nanoTime();\n      byte[] outputBytes = Files.readAllBytes(tmpFile);\n      long readLatency = TimeUnit.MICROSECONDS.convert(\n          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);\n      metric.addReadFileLatency(readLatency);\n\n      // validation\n      if (!Arrays.equals(inputBytes, outputBytes)) {\n        metric.diskCheckFailed();\n        throw new DiskErrorException(\"Data in file has been corrupted.\");\n      }\n    } catch (IOException e) {\n      metric.diskCheckFailed();\n      throw new DiskErrorException(\"Disk Check failed!\", e);\n    } finally {\n      // delete the file\n      if (tmpFile != null) {\n        try {\n          Files.delete(tmpFile);\n        } catch (IOException e) {\n          metric.diskCheckFailed();\n          throw new DiskErrorException(\"File deletion failed!\", e);\n        }\n      }\n    }\n  }\n}\n","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java#L60-L96","documentation":"The core integrity probe of ReadWriteDiskValidator: it writes 16 random bytes to a fresh temp file in the target directory, reads them back, and compares with Arrays.equals. A mismatch throws DiskErrorException(\"Data in file has been corrupted.\") — the storage returned bytes different from what was written, a strong signal of failing hardware or a broken filesystem layer rather than a configuration problem.","triggerScenarios":"checkStatus on a volume whose reads do not return what was written: dying disk, faulty cable/controller/NVMe, buggy NFS/CIFS caching layers, severe memory errors corrupting data before it reaches the platter.","commonSituations":"NodeManager local volumes degrading on aging worker nodes; cheap drives in dense clusters; network filesystems with coherence bugs; the error recurs every disk-check interval until the volume is removed or repaired.","solutions":["Inspect dmesg/syslog for I/O errors on that device, then run smartctl -a and fsck","Drain the node and replace the failing disk, or drop the directory from local-dirs until repaired","If the volume is network-attached, reproduce locally to rule out the mount layer","After replacement, confirm ReadWriteDiskValidatorMetrics failure counters stop rising"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { diskValidator.checkStatus(dir); } catch (DiskErrorException e) { volume.markFailed(); ops.alert(\"possible disk corruption on \" + dir + \": \" + e.getMessage()); }","preventionTips":["Monitor ReadWriteDiskValidatorMetrics failure counters and alert on first occurrence","Keep SMART monitoring enabled on worker nodes","Investigate the very first corruption event — repeats mean failing hardware"],"tags":["hadoop","yarn","disk-check","hardware","data-corruption"],"backgroundTag":"disk-data-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}