apache/hadoop · error · IOException

Checksum verification failed for the block ${blockFileName}:

Error message

Checksum verification failed for the block ${blockFileName}: premature EOF

What it means

Thrown by NativePmemMappableBlockLoader.verifyChecksumAndMapBlock() (NativePmemMappableBlockLoader.java:153): while streaming the block through its channel for checksum verification before publishing the pmem mapping, fillBuffer(blockChannel, blockBuf) hit EOF before 'length' bytes. The block file on disk is shorter than the length the DataNode intends to cache - a truncated or corrupted block replica.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java:153

      DataChecksum checksum = header.getChecksum();
      final int bytesPerChecksum = checksum.getBytesPerChecksum();
      final int checksumSize = checksum.getChecksumSize();
      final int numChunks = (8 * 1024 * 1024) / bytesPerChecksum;
      ByteBuffer blockBuf = ByteBuffer.allocate(numChunks * bytesPerChecksum);
      ByteBuffer checksumBuf = ByteBuffer.allocate(numChunks * checksumSize);
      // Verify the checksum
      int bytesVerified = 0;
      long mappedAddress = -1L;
      if (region != null) {
        mappedAddress = region.getAddress();
      }
      while (bytesVerified < length) {
        Preconditions.checkState(bytesVerified % bytesPerChecksum == 0,
            "Unexpected partial chunk before EOF.");
        assert bytesVerified % bytesPerChecksum == 0;
        int bytesRead = fillBuffer(blockChannel, blockBuf);
        if (bytesRead == -1) {
          throw new IOException(
              "Checksum verification failed for the block " + blockFileName +
                  ": premature EOF");
        }
        blockBuf.flip();
        // Number of read chunks, including partial chunk at end
        int chunks = (bytesRead + bytesPerChecksum - 1) / bytesPerChecksum;
        checksumBuf.limit(chunks * checksumSize);
        fillBuffer(metaChannel, checksumBuf);
        checksumBuf.flip();
        checksum.verifyChunkedSums(blockBuf, checksumBuf, blockFileName,
            bytesVerified);
        // Success
        bytesVerified += bytesRead;
        // Copy data to persistent file
        POSIX.Pmem.memCopy(blockBuf.array(), mappedAddress,
            region.isPmem(), bytesRead);
        mappedAddress += bytesRead;
        // Clear buffer

View on GitHub (pinned to 2add963021)

Solutions

hdfs fsck the affected path, confirm the damaged replica, and delete it so the NameNode re-replicates from a healthy copy; the cache directive retries after repair
Compare the block file size with the NameNode's block length to confirm truncation
Check disk health on that volume (smartctl, dmesg) and retire it if errors cluster there
After the replica is re-replicated, re-run the cache directive or wait for the automatic retry

Example fix

// before: verifying while caching a possibly-short replica
loader.load(length, blockIn, metaIn, blockFileName, key);

// after: pre-check sizes so short replicas fail fast with a clear cause
if (blockFile.length() < length || metaFile.length() <
    BlockMetadataHeader.getHeaderSize()
    + ((length + bytesPerChecksum - 1) / bytesPerChecksum) * checksumSize) {
  throw new IOException("Truncated replica " + blockFile
      + ": block=" + blockFile.length() + "/" + length
      + ", meta=" + metaFile.length());
}
loader.load(length, blockIn, metaIn, blockFileName, key);

Defensive patterns

Strategy: validation

Validate before calling

// skip caching replicas whose block file cannot satisfy the length
if (blockFile.length() < length) {
  LOG.warn("Not caching truncated replica {}: {}/{} bytes",
      blockFile, blockFile.length(), length);
  return;
}

Try / catch

try {
  loader.load(length, blockIn, metaIn, blockFileName, key);
} catch (IOException e) {
  // premature EOF => truncated replica: invalidate it, fsck the file, let re-replication fix it
  LOG.warn("pmem cache verify hit EOF for {}: {}", blockFileName, e);
}

Prevention

fsck datasets after disk incidents before pointing cache directives at them
Compare block file sizes to NN-recorded lengths during routine volume scans
Retire volumes that produce short reads instead of repeatedly failing cache attempts

When it happens

Trigger: A cache directive on a replica whose block file was truncated by disk corruption, partial restore, or external modification; short reads from a failing disk; block registered with a length larger than the file actually present.

Common situations: Caching datasets that sit on a marginal/failing volume; block files touched by backup tooling or restored inconsistently; pmem cache verification exposing corruption that reads had not yet noticed.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b64bf73dcd5c3062. Report an issue: GitHub.