{"record":{"id":"f421d3c5dfab88e4","repo":"apache/hadoop","slug":"checksum-verification-failed-premature-eof","errorCode":null,"errorMessage":"checksum verification failed: premature EOF","messagePattern":"checksum verification failed: premature EOF","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java","lineNumber":157,"sourceCode":"      if (metaChannel == null) {\n        throw new IOException(\n            \"Block InputStream meta file has no FileChannel.\");\n      }\n      DataChecksum checksum = header.getChecksum();\n      final int bytesPerChecksum = checksum.getBytesPerChecksum();\n      final int checksumSize = checksum.getChecksumSize();\n      final int numChunks = (8 * 1024 * 1024) / bytesPerChecksum;\n      ByteBuffer blockBuf = ByteBuffer.allocate(numChunks * bytesPerChecksum);\n      ByteBuffer checksumBuf = ByteBuffer.allocate(numChunks * checksumSize);\n      // Verify the checksum\n      int bytesVerified = 0;\n      while (bytesVerified < length) {\n        Preconditions.checkState(bytesVerified % bytesPerChecksum == 0,\n            \"Unexpected partial chunk before EOF\");\n        assert bytesVerified % bytesPerChecksum == 0;\n        int bytesRead = fillBuffer(blockChannel, blockBuf);\n        if (bytesRead == -1) {\n          throw new IOException(\"checksum verification failed: premature EOF\");\n        }\n        blockBuf.flip();\n        // Number of read chunks, including partial chunk at end\n        int chunks = (bytesRead + bytesPerChecksum - 1) / bytesPerChecksum;\n        checksumBuf.limit(chunks * checksumSize);\n        fillBuffer(metaChannel, checksumBuf);\n        checksumBuf.flip();\n        checksum.verifyChunkedSums(blockBuf, checksumBuf, blockFileName,\n            bytesVerified);\n        // Success\n        bytesVerified += bytesRead;\n        blockBuf.clear();\n        checksumBuf.clear();\n      }\n    }\n  }\n\n  /**","sourceCodeStart":139,"sourceCodeEnd":175,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java#L139-L175","documentation":"Thrown by MappableBlockLoader.verifyChecksum() (MappableBlockLoader.java:157) during centralized-cache load: fillBuffer(blockChannel, blockBuf) hit EOF before 'length' bytes of the block file were read and verified. The expected length (from the replica being cached) is larger than the actual block file, i.e. the block file is truncated/corrupted relative to what the DataNode believes it stores.","triggerScenarios":"hdfs cacheadmin -addDirective triggering a cache of a replica whose block file was truncated by disk corruption or a partial restore; block file length changed (manual intervention, bit rot) after the replica was registered; a block file on a failing volume returning short reads.","commonSituations":"Caching files that live on a marginal disk; volumes restored from inconsistent backups; verifying caches right after disk errors appear in dmesg; block files shrunk by external tooling.","solutions":["Run hdfs fsck <path> -files -blocks to identify the damaged replica; delete it so the NameNode re-replicates from healthy peers, then re-apply the cache directive","Check volume health (smartctl, dmesg) - truncation on one volume usually warrants retiring it","Compare block file size against the NameNode's recorded block length to confirm which side is wrong","After repair, the failed caching task retries automatically on the next cache report round"],"exampleFix":"// before: caching without validating the replica first\ntry (FileInputStream blockIn = fileIoProvider.getFileInputStream(vol, blockFile)) {\n  loader.load(length, blockIn, metaIn, blockFileName, key);\n}\n\n// after: skip corrupt replicas so caching targets a healthy copy\nif (blockFile.length() < length) {\n  LOG.warn(\"Block file {} shorter than expected {}; skipping cache\",\n      blockFile, length);\n  return;\n}\ntry (FileInputStream blockIn = fileIoProvider.getFileInputStream(vol, blockFile)) {\n  loader.load(length, blockIn, metaIn, blockFileName, key);\n}","handlingStrategy":"validation","validationCode":"// only cache replicas whose block file can possibly satisfy the length\nif (blockFile.length() < length) {\n  LOG.warn(\"Block file {} is {} bytes, expected {}; not caching corrupt replica\",\n      blockFile, blockFile.length(), length);\n  return;\n}","typeGuard":null,"tryCatchPattern":"try {\n  loader.load(length, blockIn, metaIn, blockFileName, key);\n} catch (IOException e) {\n  // premature EOF => truncated replica: fsck + invalidate the replica, then let the directive re-cache\n  LOG.warn(\"Cache verify failed for {}: {}\", blockFileName, e);\n}","preventionTips":["Validate replica sizes before adding cache directives over recovered/restored data","Monitor disk health; truncated block files usually ride on failing hardware","After any restore-from-backup event, run fsck before caching those files"],"tags":["hdfs","datanode","centralized-cache","checksum","corruption"],"backgroundTag":"checksum-verification-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}