{"record":{"id":"b64bf73dcd5c3062","repo":"apache/hadoop","slug":"checksum-verification-failed-for-the-block-block","errorCode":null,"errorMessage":"Checksum verification failed for the block ${blockFileName}: premature EOF","messagePattern":"Checksum verification failed for the block (.+?): premature EOF","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java","lineNumber":153,"sourceCode":"      DataChecksum checksum = header.getChecksum();\n      final int bytesPerChecksum = checksum.getBytesPerChecksum();\n      final int checksumSize = checksum.getChecksumSize();\n      final int numChunks = (8 * 1024 * 1024) / bytesPerChecksum;\n      ByteBuffer blockBuf = ByteBuffer.allocate(numChunks * bytesPerChecksum);\n      ByteBuffer checksumBuf = ByteBuffer.allocate(numChunks * checksumSize);\n      // Verify the checksum\n      int bytesVerified = 0;\n      long mappedAddress = -1L;\n      if (region != null) {\n        mappedAddress = region.getAddress();\n      }\n      while (bytesVerified < length) {\n        Preconditions.checkState(bytesVerified % bytesPerChecksum == 0,\n            \"Unexpected partial chunk before EOF.\");\n        assert bytesVerified % bytesPerChecksum == 0;\n        int bytesRead = fillBuffer(blockChannel, blockBuf);\n        if (bytesRead == -1) {\n          throw new IOException(\n              \"Checksum verification failed for the block \" + blockFileName +\n                  \": premature EOF\");\n        }\n        blockBuf.flip();\n        // Number of read chunks, including partial chunk at end\n        int chunks = (bytesRead + bytesPerChecksum - 1) / bytesPerChecksum;\n        checksumBuf.limit(chunks * checksumSize);\n        fillBuffer(metaChannel, checksumBuf);\n        checksumBuf.flip();\n        checksum.verifyChunkedSums(blockBuf, checksumBuf, blockFileName,\n            bytesVerified);\n        // Success\n        bytesVerified += bytesRead;\n        // Copy data to persistent file\n        POSIX.Pmem.memCopy(blockBuf.array(), mappedAddress,\n            region.isPmem(), bytesRead);\n        mappedAddress += bytesRead;\n        // Clear buffer","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java#L135-L171","documentation":"Thrown by NativePmemMappableBlockLoader.verifyChecksumAndMapBlock() (NativePmemMappableBlockLoader.java:153): while streaming the block through its channel for checksum verification before publishing the pmem mapping, fillBuffer(blockChannel, blockBuf) hit EOF before 'length' bytes. The block file on disk is shorter than the length the DataNode intends to cache - a truncated or corrupted block replica.","triggerScenarios":"A cache directive on a replica whose block file was truncated by disk corruption, partial restore, or external modification; short reads from a failing disk; block registered with a length larger than the file actually present.","commonSituations":"Caching datasets that sit on a marginal/failing volume; block files touched by backup tooling or restored inconsistently; pmem cache verification exposing corruption that reads had not yet noticed.","solutions":["hdfs fsck the affected path, confirm the damaged replica, and delete it so the NameNode re-replicates from a healthy copy; the cache directive retries after repair","Compare the block file size with the NameNode's block length to confirm truncation","Check disk health on that volume (smartctl, dmesg) and retire it if errors cluster there","After the replica is re-replicated, re-run the cache directive or wait for the automatic retry"],"exampleFix":"// before: verifying while caching a possibly-short replica\nloader.load(length, blockIn, metaIn, blockFileName, key);\n\n// after: pre-check sizes so short replicas fail fast with a clear cause\nif (blockFile.length() < length || metaFile.length() <\n    BlockMetadataHeader.getHeaderSize()\n    + ((length + bytesPerChecksum - 1) / bytesPerChecksum) * checksumSize) {\n  throw new IOException(\"Truncated replica \" + blockFile\n      + \": block=\" + blockFile.length() + \"/\" + length\n      + \", meta=\" + metaFile.length());\n}\nloader.load(length, blockIn, metaIn, blockFileName, key);","handlingStrategy":"validation","validationCode":"// skip caching replicas whose block file cannot satisfy the length\nif (blockFile.length() < length) {\n  LOG.warn(\"Not caching truncated replica {}: {}/{} bytes\",\n      blockFile, blockFile.length(), length);\n  return;\n}","typeGuard":null,"tryCatchPattern":"try {\n  loader.load(length, blockIn, metaIn, blockFileName, key);\n} catch (IOException e) {\n  // premature EOF => truncated replica: invalidate it, fsck the file, let re-replication fix it\n  LOG.warn(\"pmem cache verify hit EOF for {}: {}\", blockFileName, e);\n}","preventionTips":["fsck datasets after disk incidents before pointing cache directives at them","Compare block file sizes to NN-recorded lengths during routine volume scans","Retire volumes that produce short reads instead of repeatedly failing cache attempts"],"tags":["hdfs","datanode","pmem","centralized-cache","checksum","corruption"],"backgroundTag":"checksum-verification-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}