apache/hadoop · error · IOException

Got unexpected checksum file EOF at {}, block file position

Error message

Got unexpected checksum file EOF at {}, block file position {} for block {} of file {}

What it means

BlockReaderLocal performs short-circuit reads directly from the datanode's block and .meta files, verifying checksums locally. Here it computed exactly how many checksum bytes the .meta file must contain for the data range being read, and read() on the metadata stream returned EOF before that many bytes arrived — the checksum file is shorter than the data file implies, i.e., the local replica is corrupt or truncated on disk.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderLocal.java:378

    if (canSkipChecksum) {
      freeChecksumBufIfExists();
      return total;
    }
    if (total > 0) {
      try {
        buf.limit(buf.position());
        buf.position(startBufPos);
        createChecksumBufIfNeeded();
        int checksumsNeeded = (total + bytesPerChecksum - 1) /
            bytesPerChecksum;
        checksumBuf.clear();
        checksumBuf.limit(checksumsNeeded * checksumSize);
        long checksumPos = BlockMetadataHeader.getHeaderSize()
            + ((startDataPos / bytesPerChecksum) * checksumSize);
        while (checksumBuf.hasRemaining()) {
          int nRead = checksumIn.read(checksumBuf, checksumPos);
          if (nRead < 0) {
            throw new IOException("Got unexpected checksum file EOF at " +
                checksumPos + ", block file position " + startDataPos +
                " for block " + block + " of file " + filename);
          }
          checksumPos += nRead;
        }
        checksumBuf.flip();

        checksum.verifyChunkedSums(buf, checksumBuf, filename, startDataPos);
      } finally {
        buf.position(buf.limit());
      }
    }
    return total;
  }

  private boolean createNoChecksumContext() {
    return !verifyChecksum ||
        // Checksums are not stored for replicas on transient storage.  We do

View on GitHub (pinned to 2add963021)

Solutions

1. Run 'hdfs fsck /path -files -blocks -locations' to confirm which replica is corrupt.
2. Force re-replication of the bad replica: stop the affected datanode briefly, or use 'hdfs fsck -delete'/'-move' once corruption is confirmed.
3. As an immediate workaround set dfs.client.read.shortcircuit=false so clients read a healthy remote replica.
4. Inspect the datanode host: dmesg/SMART errors, full disks, and datanode logs for block write failures.

Example fix

# diagnose
hdfs fsck /path/to/file -files -blocks -locations
# heal once corruption is confirmed (drops corrupt replicas, triggers re-replication)
hdfs fsck /path/to/file -delete

try {
  return readLocal(dfs, path); // short-circuit path
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("checksum file EOF")) {
    // local replica is corrupt: read a remote replica and alert ops
    conf.setBoolean("dfs.client.read.shortcircuit", false);
    alertOps("Possible corrupt replica for " + path);
    return readNetwork(dfs, path);
  }
  throw e;
}

When it happens

Trigger: A short-circuit read of a block whose .meta (checksum) file is truncated relative to the block data — after disk-full crashes, partial writes, or on-disk corruption. The client is co-located with that datanode, which is why only some nodes see it.

Common situations: Failing disks on datanodes; interrupted block writes leaving inconsistent block/meta pairs; corruption that surfaces only after short-circuit reads are enabled because remote reads pick other replicas.

Related errors

• Meta file for {} not found.
• Configured BlockReaderLocalLegacy buffer size ({}) is not la
• Failed to move meta file for {b} from {metadataURI} to {dstm
• Failed to copy {srcReplica} metadata to {dstMeta}
• Failed to hardLink {srcReplica} metadata to {dstMeta}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c240921065e716e1. Report an issue: GitHub.