apache/hadoop · warning · ReplicaNotFoundException

Replica not found for {b}. The block may have been removed r

Error message

Replica not found for {b}. The block may have been removed recently by the balancer or by intentionally reducing the replication factor. This condition is usually harmless. To be certain, please check the preceding datanode log messages for signs of a more serious issue.

What it means

ReplicaNotFoundException (message appends POSSIBLE_ROOT_CAUSE_MSG about the balancer / replication-factor reduction) thrown by FsDatasetImpl.checkBlock when volumeMap has no entry for the block's (blockPoolId, localBlock). The exception text itself states this condition is usually harmless: the block was most likely deleted from this DataNode after the caller obtained a stale block location.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2257

   *
   * @throws ReplicaNotFoundException          If the replica is not found 
   *
   * @throws UnexpectedReplicaStateException   If the replica is not in the 
   *                                             expected state.
   * @throws FileNotFoundException             If the block file is not found or there
   *                                              was an error locating it.
   * @throws EOFException                      If the replica length is too short.
   * 
   * @throws IOException                       May be thrown from the methods called. 
   */
  @Override // FsDatasetSpi
  public void checkBlock(ExtendedBlock b, long minLength, ReplicaState state)
      throws ReplicaNotFoundException, UnexpectedReplicaStateException,
      FileNotFoundException, EOFException, IOException {
    final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(), 
        b.getLocalBlock());
    if (replicaInfo == null) {
      throw new ReplicaNotFoundException(b);
    }
    if (replicaInfo.getState() != state) {
      throw new UnexpectedReplicaStateException(b,state);
    }
    if (!replicaInfo.blockDataExists()) {
      throw new FileNotFoundException(replicaInfo.getBlockURI().toString());
    }
    long onDiskLength = getLength(b);
    if (onDiskLength < minLength) {
      throw new EOFException(b + "'s on-disk length " + onDiskLength
          + " is shorter than minLength " + minLength);
    }
  }

  /**
   * Check whether the given block is a valid one.
   * valid means finalized
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Treat as expected churn: the reader will retry against other replicas returned by the NameNode.
2. If clients keep failing, refresh their block locations (open a new DFSInputStream instead of reusing a stale one).
3. Confirm with the NameNode that replication of the block is healthy (hdfs fsck /path -files -blocks).
4. Check preceding DataNode logs - as the message advises - to rule out disk errors that silently removed replicas.

// Pre-check presence without exceptions:
ReplicaInfo r = dataset.getReplicaInfo(b); // null-safe lookup where exposed
if (r == null) {
  // treat as replica churn: fall back to other locations in LocatedBlocks
}

boolean isReplicaNotFound(IOException e) {
  return e instanceof org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException;
}

try {
  dataset.checkBlock(b, minLength, ReplicaState.FINALIZED);
} catch (ReplicaNotFoundException e) {
  // expected churn (balancer / setrep): pick the next replica from LocatedBlocks and retry
  return nextReplica(b);
}

When it happens

Trigger: checkBlock(b, minLength, state) called with an ExtendedBlock whose block id is no longer in the volumeMap - e.g., a read/verification request routed to a replica that was invalidated between NameNode block-report and DataNode lookup.

Common situations: Balancer moved the replica; admin lowered replication factor (hadoop fs -setrep) so excess replicas were deleted; NameNode block report lag right after deletion; client cached an old LocatedBlock.

Related errors

• Replica not found for {block}. The block may have been remov
• Replica does not exist {}
• Replica does not exist {}:{}
• Replica does not exist {b}
• Replica {b} is not in expected state {state}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/651dfd910d9087a8. Report an issue: GitHub.