apache/hadoop · error · ChecksumException

Checksum failed at {failedPos} for replica: {replica}

Error message

Checksum failed at {failedPos} for replica: {replica}

What it means

While serving chunks with verifyChecksum enabled, BlockSender recomputes each chunk's checksum and compares it against the stored checksum in the meta file (checksum.compare(buf, cOff)). A mismatch means the bytes on disk no longer match their recorded checksum, so ChecksumException is thrown carrying failedPos (offset + datalen - dLeft, the exact byte position of the first bad chunk) and the replica details — silent data corruption has been detected on this replica.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:739

   */
  public void verifyChecksum(final byte[] buf, final int dataOffset,
      final int datalen, final int numChunks, final int checksumOffset)
      throws ChecksumException {
    int dOff = dataOffset;
    int cOff = checksumOffset;
    int dLeft = datalen;

    for (int i = 0; i < numChunks; i++) {
      checksum.reset();
      int dLen = Math.min(dLeft, chunkSize);
      checksum.update(buf, dOff, dLen);
      if (!checksum.compare(buf, cOff)) {
        long failedPos = offset + datalen - dLeft;
        StringBuilder replicaInfoString = new StringBuilder();
        if (replica != null) {
          replicaInfoString.append(" for replica: " + replica.toString());
        }
        throw new ChecksumException("Checksum failed at " + failedPos
            + replicaInfoString, failedPos);
      }
      dLeft -= dLen;
      dOff += dLen;
      cOff += checksumSize;
    }
  }
  
  /**
   * sendBlock() is used to read block and its metadata and stream the data to
   * either a client or to another datanode. 
   * 
   * @param out  stream to which the block is written to
   * @param baseStream optional. if non-null, <code>out</code> is assumed to 
   *        be a wrapper over this stream. This enables optimizations for
   *        sending the data, e.g. 
   *        {@link SocketOutputStream#transferToFully(FileChannel, 
   *        long, int)}.

View on GitHub (pinned to 2add963021)

Solutions

1. Let the read fail over: DFSClient transparently tries another replica, so client impact is limited unless all replicas are corrupt.
2. Report/remove the corrupt replica: the block scanner or an fsck-triggered report marks it corrupt and the NN schedules re-replication from a good copy.
3. If multiple replicas fail checksums, restore the file from backup/snapshot — the data is genuinely lost.
4. Harden the stack: enable dfs.datanode.scan.period.hours, use ECC RAM, and prefer drives/paths with data-integrity protection.

static boolean isChecksumFailure(IOException e) {
  return e instanceof ChecksumException;
}

try {
  in.read(buf, off, len);
} catch (ChecksumException e) {
  reportCorruptBlockToNameNode(e); // includes failedPos from the exception
  in = reopenFromNextReplica();    // DFSInputStream-style failover
  in.read(buf, off, len);
}

When it happens

Trigger: readBlock with checksum verification on data whose on-disk bytes diverge from the .meta checksums: bit rot on the disk, corruption introduced by a bad disk controller/RAM, or a block file modified out-of-band. Any subsequent read of that chunk throws at failedPos.

Common situations: Bit rot on consumer drives without T10-DIF/end-to-end checksums; faulty cables/controllers writing garbage; memory errors corrupting data before write; the datanode block scanner finding the same corruption in the background.

Related errors

• Meta-data not found for {block}
• File " + url + " computed digest " + computedDigest + " does
• Replica gen stamp < block genstamp, block={block}, replica={
• Replica is not readable, block={block}, replica={replica}
• Replica not found for {block}. The block may have been remov

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2b874c322cbde93c. Report an issue: GitHub.