{"record":{"id":"ee70c7a09fb1d499","repo":"apache/hadoop","slug":"cannot-recover-append-close-to-a-replica-that-s-no","errorCode":null,"errorMessage":"Cannot recover append/close to a replica that's not FINALIZED and not RBW {replicaInfo}","messagePattern":"Cannot recover append/close to a replica that's not FINALIZED and not RBW (.+?)","errorType":"exception","errorClass":"ReplicaNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1507,"sourceCode":"    private final ReplicaInPipeline rip;\n\n    MustStopExistingWriter(ReplicaInPipeline rip) {\n      this.rip = rip;\n    }\n\n    ReplicaInPipeline getReplicaInPipeline() {\n      return rip;\n    }\n  }\n\n  private ReplicaInfo recoverCheck(ExtendedBlock b, long newGS, \n      long expectedBlockLen) throws IOException, MustStopExistingWriter {\n    ReplicaInfo replicaInfo = getReplicaInfo(b.getBlockPoolId(), b.getBlockId());\n    \n    // check state\n    if (replicaInfo.getState() != ReplicaState.FINALIZED &&\n        replicaInfo.getState() != ReplicaState.RBW) {\n      throw new ReplicaNotFoundException(\n          ReplicaNotFoundException.UNFINALIZED_AND_NONRBW_REPLICA + replicaInfo);\n    }\n\n    // check generation stamp\n    long replicaGenerationStamp = replicaInfo.getGenerationStamp();\n    if (replicaGenerationStamp < b.getGenerationStamp() ||\n        replicaGenerationStamp > newGS) {\n      throw new ReplicaNotFoundException(\n          ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + replicaGenerationStamp\n          + \". Expected GS range is [\" + b.getGenerationStamp() + \", \" + \n          newGS + \"].\");\n    }\n    \n    // stop the previous writer before check a replica's length\n    long replicaLen = replicaInfo.getNumBytes();\n    if (replicaInfo.getState() == ReplicaState.RBW) {\n      ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;\n      if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {","sourceCodeStart":1489,"sourceCodeEnd":1525,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1489-L1525","documentation":"recoverCheck() is shared by recoverAppend and recoverClose and accepts only FINALIZED or RBW replicas. Any other state - RWR (replica waiting for recovery, created when a DataNode restarts while a block is being written) or TEMPORARY - is rejected with ReplicaNotFoundException (UNFINALIZED_AND_NONRBW_REPLICA), because neither state can legally continue an append or close.","triggerScenarios":"Lease/block recovery reaches this DataNode while its replica is RWR (DataNode restarted mid-write and re-registered the replica) or TEMPORARY; recoverAppend(b, newGS, expectedBlockLen) or recoverClose then fails validation before any truncation happens.","commonSituations":"Lease recovery racing a DataNode restart: the NN still lists the DN but its replica re-registered as RWR; recovery retried by the NN usually succeeds once states settle; persistent failures when a replica is wedged in an unexpected on-disk state.","solutions":["Retry and let the NameNode re-run block/lease recovery - after re-registration the replica transitions to a recoverable state or is invalidated","Check DataNode logs that re-registration with the block pool succeeded and no storage directory failed","If the replica stays RWR indefinitely, restart the DataNode or remove the replica so it is re-replicated","Use 'hdfs fsck -openforwrite' to list files whose recovery is stuck"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nReplicaState s = (r != null) ? r.getState() : null;\nif (s != ReplicaState.FINALIZED && s != ReplicaState.RBW) {\n  // RWR/TEMPORARY: wait for NN-driven recovery and re-registration; do not call now\n  scheduleRecoveryRetry();\n  return;\n}\nfsDataset.recoverAppend(b, newGS, expectedBlockLen);","typeGuard":"boolean isRecoverable(Replica r) {\n  return r != null\n      && (r.getState() == ReplicaState.FINALIZED || r.getState() == ReplicaState.RBW);\n}","tryCatchPattern":"catch (ReplicaNotFoundException rnfe) {\n  if (rnfe.getMessage().contains(ReplicaNotFoundException.UNFINALIZED_AND_NONRBW_REPLICA)) {\n    awaitDatanodeReRegistrationThenRetryRecoveryOnce();\n  } else { throw rnfe; }\n}","preventionTips":["After a DataNode restart, let it re-register and reconcile replicas before initiating recovery on its blocks","Monitor DataNode registration and storage-directory errors - wedged replicas start there","Close files cleanly so recovery never has to touch non-standard replica states"],"tags":["hdfs","datanode","block-recovery","replica-state","datanode-restart"],"backgroundTag":"hdfs-replica-state-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}