What causes "checksum verification failed: premature EOF" in apache/hadoop?

Run hdfs fsck -files -blocks to identify the damaged replica; delete it so the NameNode re-replicates from healthy peers, then re-apply the cache directive

apache/hadoop · error · IOException

checksum verification failed: premature EOF

Error message

checksum verification failed: premature EOF

What it means

Thrown by MappableBlockLoader.verifyChecksum() (MappableBlockLoader.java:157) during centralized-cache load: fillBuffer(blockChannel, blockBuf) hit EOF before 'length' bytes of the block file were read and verified. The expected length (from the replica being cached) is larger than the actual block file, i.e. the block file is truncated/corrupted relative to what the DataNode believes it stores.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoader.java:157

      if (metaChannel == null) {
        throw new IOException(
            "Block InputStream meta file has no FileChannel.");
      }
      DataChecksum checksum = header.getChecksum();
      final int bytesPerChecksum = checksum.getBytesPerChecksum();
      final int checksumSize = checksum.getChecksumSize();
      final int numChunks = (8 * 1024 * 1024) / bytesPerChecksum;
      ByteBuffer blockBuf = ByteBuffer.allocate(numChunks * bytesPerChecksum);
      ByteBuffer checksumBuf = ByteBuffer.allocate(numChunks * checksumSize);
      // Verify the checksum
      int bytesVerified = 0;
      while (bytesVerified < length) {
        Preconditions.checkState(bytesVerified % bytesPerChecksum == 0,
            "Unexpected partial chunk before EOF");
        assert bytesVerified % bytesPerChecksum == 0;
        int bytesRead = fillBuffer(blockChannel, blockBuf);
        if (bytesRead == -1) {
          throw new IOException("checksum verification failed: premature EOF");
        }
        blockBuf.flip();
        // Number of read chunks, including partial chunk at end
        int chunks = (bytesRead + bytesPerChecksum - 1) / bytesPerChecksum;
        checksumBuf.limit(chunks * checksumSize);
        fillBuffer(metaChannel, checksumBuf);
        checksumBuf.flip();
        checksum.verifyChunkedSums(blockBuf, checksumBuf, blockFileName,
            bytesVerified);
        // Success
        bytesVerified += bytesRead;
        blockBuf.clear();
        checksumBuf.clear();
      }
    }
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Run hdfs fsck <path> -files -blocks to identify the damaged replica; delete it so the NameNode re-replicates from healthy peers, then re-apply the cache directive
Check volume health (smartctl, dmesg) - truncation on one volume usually warrants retiring it
Compare block file size against the NameNode's recorded block length to confirm which side is wrong
After repair, the failed caching task retries automatically on the next cache report round

Example fix

// before: caching without validating the replica first
try (FileInputStream blockIn = fileIoProvider.getFileInputStream(vol, blockFile)) {
  loader.load(length, blockIn, metaIn, blockFileName, key);
}

// after: skip corrupt replicas so caching targets a healthy copy
if (blockFile.length() < length) {
  LOG.warn("Block file {} shorter than expected {}; skipping cache",
      blockFile, length);
  return;
}
try (FileInputStream blockIn = fileIoProvider.getFileInputStream(vol, blockFile)) {
  loader.load(length, blockIn, metaIn, blockFileName, key);
}

Defensive patterns

Strategy: validation

Validate before calling

// only cache replicas whose block file can possibly satisfy the length
if (blockFile.length() < length) {
  LOG.warn("Block file {} is {} bytes, expected {}; not caching corrupt replica",
      blockFile, blockFile.length(), length);
  return;
}

Try / catch

try {
  loader.load(length, blockIn, metaIn, blockFileName, key);
} catch (IOException e) {
  // premature EOF => truncated replica: fsck + invalidate the replica, then let the directive re-cache
  LOG.warn("Cache verify failed for {}: {}", blockFileName, e);
}

Prevention

Validate replica sizes before adding cache directives over recovered/restored data
Monitor disk health; truncated block files usually ride on failing hardware
After any restore-from-backup event, run fsck before caching those files

When it happens

Trigger: hdfs cacheadmin -addDirective triggering a cache of a replica whose block file was truncated by disk corruption or a partial restore; block file length changed (manual intervention, bit rot) after the replica was registered; a block file on a failing volume returning short reads.

Common situations: Caching files that live on a marginal disk; volumes restored from inconsistent backups; verifying caches right after disk errors appear in dmesg; block files shrunk by external tooling.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f421d3c5dfab88e4. Report an issue: GitHub.