apache/hadoop · error · ReplicaNotFoundException

Cannot recover append/close to a replica that's not FINALIZE

Error message

Cannot recover append/close to a replica that's not FINALIZED and not RBW {replicaInfo}

What it means

recoverCheck() is shared by recoverAppend and recoverClose and accepts only FINALIZED or RBW replicas. Any other state - RWR (replica waiting for recovery, created when a DataNode restarts while a block is being written) or TEMPORARY - is rejected with ReplicaNotFoundException (UNFINALIZED_AND_NONRBW_REPLICA), because neither state can legally continue an append or close.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1507

    private final ReplicaInPipeline rip;

    MustStopExistingWriter(ReplicaInPipeline rip) {
      this.rip = rip;
    }

    ReplicaInPipeline getReplicaInPipeline() {
      return rip;
    }
  }

  private ReplicaInfo recoverCheck(ExtendedBlock b, long newGS, 
      long expectedBlockLen) throws IOException, MustStopExistingWriter {
    ReplicaInfo replicaInfo = getReplicaInfo(b.getBlockPoolId(), b.getBlockId());
    
    // check state
    if (replicaInfo.getState() != ReplicaState.FINALIZED &&
        replicaInfo.getState() != ReplicaState.RBW) {
      throw new ReplicaNotFoundException(
          ReplicaNotFoundException.UNFINALIZED_AND_NONRBW_REPLICA + replicaInfo);
    }

    // check generation stamp
    long replicaGenerationStamp = replicaInfo.getGenerationStamp();
    if (replicaGenerationStamp < b.getGenerationStamp() ||
        replicaGenerationStamp > newGS) {
      throw new ReplicaNotFoundException(
          ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + replicaGenerationStamp
          + ". Expected GS range is [" + b.getGenerationStamp() + ", " + 
          newGS + "].");
    }
    
    // stop the previous writer before check a replica's length
    long replicaLen = replicaInfo.getNumBytes();
    if (replicaInfo.getState() == ReplicaState.RBW) {
      ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;
      if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {

View on GitHub (pinned to 2add963021)

Solutions

1. Retry and let the NameNode re-run block/lease recovery - after re-registration the replica transitions to a recoverable state or is invalidated
2. Check DataNode logs that re-registration with the block pool succeeded and no storage directory failed
3. If the replica stays RWR indefinitely, restart the DataNode or remove the replica so it is re-replicated
4. Use 'hdfs fsck -openforwrite' to list files whose recovery is stuck

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
ReplicaState s = (r != null) ? r.getState() : null;
if (s != ReplicaState.FINALIZED && s != ReplicaState.RBW) {
  // RWR/TEMPORARY: wait for NN-driven recovery and re-registration; do not call now
  scheduleRecoveryRetry();
  return;
}
fsDataset.recoverAppend(b, newGS, expectedBlockLen);

boolean isRecoverable(Replica r) {
  return r != null
      && (r.getState() == ReplicaState.FINALIZED || r.getState() == ReplicaState.RBW);
}

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage().contains(ReplicaNotFoundException.UNFINALIZED_AND_NONRBW_REPLICA)) {
    awaitDatanodeReRegistrationThenRetryRecoveryOnce();
  } else { throw rnfe; }
}

When it happens

Trigger: Lease/block recovery reaches this DataNode while its replica is RWR (DataNode restarted mid-write and re-registered the replica) or TEMPORARY; recoverAppend(b, newGS, expectedBlockLen) or recoverClose then fails validation before any truncation happens.

Common situations: Lease recovery racing a DataNode restart: the NN still lists the DN but its replica re-registered as RWR; recovery retried by the NN usually succeeds once states settle; persistent failures when a replica is wedged in an unexpected on-disk state.

Related errors

• Cannot recover a non-RBW replica {replicaInfo}
• Expected replica state: RBW obtained {state} for converting
• replica.getState() != RUR, replica={replica}
• Replica was found but missing fields.
• {} has no enough internal blocks(current: {}), unable to sta

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ee70c7a09fb1d499. Report an issue: GitHub.