apache/hadoop · error · IOException

fetchBlockByteRange(). Got a checksum exception for {} at {}

Error message

fetchBlockByteRange(). Got a checksum exception for {} at {}:{} from {}

What it means

The DataNode raised a ChecksumException (CRC mismatch) for the requested range. The client logs it, records block+DN in corruptedBlocks (later reported to the NameNode as a bad block), adds the DN to the local dead list, and rethrows the message as a plain IOException so the outer loop retries a different replica. Persistent occurrences on all replicas indicate real on-disk corruption.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1260

        IOUtilsClient.updateReadStatistics(readStatistics, nread, reader);
        dfsClient.updateFileSystemReadStats(
            reader.getNetworkDistance(), nread, readTimeMS);
        if (nread != len) {
          throw new IOException("truncated return from reader.read(): " +
              "excpected " + len + ", got " + nread);
        }
        DFSClientFaultInjector.get().readFromDatanodeDelay();
        return;
      } catch (ChecksumException e) {
        String msg = "fetchBlockByteRange(). Got a checksum exception for "
            + src + " at " + block.getBlock() + ":" + e.getPos() + " from "
            + datanode.info;
        DFSClient.LOG.warn(msg);
        // we want to remember what we have tried
        corruptedBlocks.addCorruptedBlock(block.getBlock(), datanode.info);
        addToLocalDeadNodes(datanode.info);
        throw new IOException(msg);
      } catch (IOException e) {
        checkInterrupted(e);
        if (e instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
          DFSClient.LOG.info("Will fetch a new encryption key and retry, "
              + "encryption key was invalid when connecting to " + datanode.addr
              + " : " + e);
          // The encryption key used is invalid.
          refetchEncryptionKey--;
          dfsClient.clearDataEncryptionKey();
        } else if (refetchToken > 0 && tokenRefetchNeeded(e, datanode.addr)) {
          refetchToken--;
          try {
            fetchBlockAt(block.getStartOffset());
          } catch (IOException fbae) {
            // ignore IOE, since we can retry it later in a loop
          }
        } else {
          String msg = String.format("Failed to read block %s for file %s from datanode %s. "

View on GitHub (pinned to 2add963021)

Solutions

1. Let the client fail over to healthy replicas (it also reports the bad block to the NN)
2. Run hdfs fsck -files -blocks -locations to list corrupt replicas
3. Remove corrupt replicas with hdfs fsck -delete or -move so the NN re-replicates from a good copy
4. If every replica is corrupt, restore the file from source/backup and replace failing disks

static boolean isChecksumFailure(IOException e) {
  return e.getMessage() != null && e.getMessage().contains("Got a checksum exception");
}

try {
  in.readFully(buf, off, len);
} catch (IOException e) {
  if (isChecksumFailure(e)) {
    // client retried other replicas and reported the bad block; surface a data-quality alert
    alertCorruptBlock(e);
  }
  throw e;
}

When it happens

Trigger: CRC of the bytes served by the DN does not match the block's checksum metadata - bit rot, truncated checksum file, or wrong-version replica after crash recovery; read retried on another replica automatically.

Common situations: Failing disks producing silent corruption; corruption introduced during unsafe cluster restarts; all replicas corrupt after long undetected disk faults (then reads fail permanently).

Related errors

• Expected to read {checksumSize} bytes from offset {offsetInC
• Expected to read {checksumSize} bytes from offset {offsetInC
• checksum verification failed: premature EOF
• Checksum verification failed for the block ${blockFileName}:
• The block meta file header is corrupt

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7367a9979346fe0e. Report an issue: GitHub.