apache/hadoop · error · ReplicaNotFoundException

Cannot recover a non-RBW replica {replicaInfo}

Error message

Cannot recover a non-RBW replica {replicaInfo}

What it means

recoverRbw is the recovery entry point for replicas that were being written. If the local replica exists but is not RBW - FINALIZED because a previous recovery already closed it, RWR after a DataNode restart, or TEMPORARY - the DataNode throws ReplicaNotFoundException with the NON_RBW_REPLICA prefix and the recovery must be routed elsewhere.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1716

  }

  @Override // FsDatasetSpi
  public ReplicaHandler recoverRbw(
      ExtendedBlock b, long newGS, long minBytesRcvd, long maxBytesRcvd)
      throws IOException {
    LOG.info("Recover RBW replica " + b);
    long startTimeMs = Time.monotonicNow();
    try {
      while (true) {
        try {
          try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
              b.getBlockPoolId(), getStorageUuidForLock(b),
              datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
            ReplicaInfo replicaInfo =
                getReplicaInfo(b.getBlockPoolId(), b.getBlockId());
            // check the replica's state
            if (replicaInfo.getState() != ReplicaState.RBW) {
              throw new ReplicaNotFoundException(
                  ReplicaNotFoundException.NON_RBW_REPLICA + replicaInfo);
            }
            ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;
            if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {
              throw new MustStopExistingWriter(rbw);
            }
            LOG.info("At " + datanode.getDisplayName() + ", Recovering " + rbw);
            return recoverRbwImpl(rbw, b, newGS, minBytesRcvd, maxBytesRcvd);
          }
        } catch (MustStopExistingWriter e) {
          e.getReplicaInPipeline().stopWriter(
              datanode.getDnConf().getXceiverStopTimeout());
        }
      }
    } finally {
      if (dataNodeMetrics != null) {
        long recoverRbwMs = Time.monotonicNow() - startTimeMs;
        dataNodeMetrics.addRecoverRbwOp(recoverRbwMs);

View on GitHub (pinned to 2add963021)

Solutions

1. Refresh block locations and rebuild the pipeline without this DN - standard DFSOutputStream retry behavior, usually self-heals
2. Retry after a pause when caused by racing recoveries
3. Run 'hdfs fsck' to confirm the block's final state (often already consistent)
4. If persistent for one DN, examine its replica state in logs and consider invalidating the replica

Example fix

// before: rebuilding the pipeline with the old DN list
client.setupPipeline(recoveredBlock, oldLocations);
// -> ReplicaNotFoundException: NON_RBW_REPLICA on the already-finalized DN

// after: refresh locations from the NN, then rebuild
LocatedBlock lb = client.getNamenode().getBlockLocations(file, 0, len).get(0);
client.setupPipeline(lb.getBlock(), lb.getLocations());

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (r == null || r.getState() != ReplicaState.RBW) {
  rebuildPipelineFromFreshLocatedBlocks(); // pick replicas whose state is RBW
  return;
}
fsDataset.recoverRbw(b, newGS, minBytesRcvd, maxBytesRcvd);

boolean isRbw(Replica r) {
  return r != null && r.getState() == ReplicaState.RBW;
}

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage().contains(ReplicaNotFoundException.NON_RBW_REPLICA)) {
    rebuildPipelineWithFreshLocations(); // often another recovery already finalized it
  } else { throw rnfe; }
}

When it happens

Trigger: BlockReceiver.java:230 recoverRbw during pipeline setup for a recovered block, when this DN's replica state has moved on: finalized by a concurrent recovery, or demoted to RWR by a DN restart between location fetch and recovery.

Common situations: Two recoveries racing (the first finalizes, the second gets NON_RBW); recovery retried against stale located blocks that still list this DN; DN restart during pipeline rebuild.

Related errors

• Replica is not readable, block={block}, replica={replica}
• Cannot recover append/close to a replica that's not FINALIZE
• RBW replica {replicaInfo}bytesRcvd({numBytes}), bytesOnDisk(
• Cannot append to a replica with unexpected generation stamp
• Unmatched length replica {rbw}: BytesAcked = {bytesAcked} By

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/63201161a41db09e. Report an issue: GitHub.