{"record":{"id":"63201161a41db09e","repo":"apache/hadoop","slug":"cannot-recover-a-non-rbw-replica-replicainfo","errorCode":null,"errorMessage":"Cannot recover a non-RBW replica {replicaInfo}","messagePattern":"Cannot recover a non-RBW replica (.+?)","errorType":"exception","errorClass":"ReplicaNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1716,"sourceCode":"  }\n\n  @Override // FsDatasetSpi\n  public ReplicaHandler recoverRbw(\n      ExtendedBlock b, long newGS, long minBytesRcvd, long maxBytesRcvd)\n      throws IOException {\n    LOG.info(\"Recover RBW replica \" + b);\n    long startTimeMs = Time.monotonicNow();\n    try {\n      while (true) {\n        try {\n          try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n              b.getBlockPoolId(), getStorageUuidForLock(b),\n              datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n            ReplicaInfo replicaInfo =\n                getReplicaInfo(b.getBlockPoolId(), b.getBlockId());\n            // check the replica's state\n            if (replicaInfo.getState() != ReplicaState.RBW) {\n              throw new ReplicaNotFoundException(\n                  ReplicaNotFoundException.NON_RBW_REPLICA + replicaInfo);\n            }\n            ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;\n            if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {\n              throw new MustStopExistingWriter(rbw);\n            }\n            LOG.info(\"At \" + datanode.getDisplayName() + \", Recovering \" + rbw);\n            return recoverRbwImpl(rbw, b, newGS, minBytesRcvd, maxBytesRcvd);\n          }\n        } catch (MustStopExistingWriter e) {\n          e.getReplicaInPipeline().stopWriter(\n              datanode.getDnConf().getXceiverStopTimeout());\n        }\n      }\n    } finally {\n      if (dataNodeMetrics != null) {\n        long recoverRbwMs = Time.monotonicNow() - startTimeMs;\n        dataNodeMetrics.addRecoverRbwOp(recoverRbwMs);","sourceCodeStart":1698,"sourceCodeEnd":1734,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1698-L1734","documentation":"recoverRbw is the recovery entry point for replicas that were being written. If the local replica exists but is not RBW - FINALIZED because a previous recovery already closed it, RWR after a DataNode restart, or TEMPORARY - the DataNode throws ReplicaNotFoundException with the NON_RBW_REPLICA prefix and the recovery must be routed elsewhere.","triggerScenarios":"BlockReceiver.java:230 recoverRbw during pipeline setup for a recovered block, when this DN's replica state has moved on: finalized by a concurrent recovery, or demoted to RWR by a DN restart between location fetch and recovery.","commonSituations":"Two recoveries racing (the first finalizes, the second gets NON_RBW); recovery retried against stale located blocks that still list this DN; DN restart during pipeline rebuild.","solutions":["Refresh block locations and rebuild the pipeline without this DN - standard DFSOutputStream retry behavior, usually self-heals","Retry after a pause when caused by racing recoveries","Run 'hdfs fsck' to confirm the block's final state (often already consistent)","If persistent for one DN, examine its replica state in logs and consider invalidating the replica"],"exampleFix":"// before: rebuilding the pipeline with the old DN list\nclient.setupPipeline(recoveredBlock, oldLocations);\n// -> ReplicaNotFoundException: NON_RBW_REPLICA on the already-finalized DN\n\n// after: refresh locations from the NN, then rebuild\nLocatedBlock lb = client.getNamenode().getBlockLocations(file, 0, len).get(0);\nclient.setupPipeline(lb.getBlock(), lb.getLocations());","handlingStrategy":"validation","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (r == null || r.getState() != ReplicaState.RBW) {\n  rebuildPipelineFromFreshLocatedBlocks(); // pick replicas whose state is RBW\n  return;\n}\nfsDataset.recoverRbw(b, newGS, minBytesRcvd, maxBytesRcvd);","typeGuard":"boolean isRbw(Replica r) {\n  return r != null && r.getState() == ReplicaState.RBW;\n}","tryCatchPattern":"catch (ReplicaNotFoundException rnfe) {\n  if (rnfe.getMessage().contains(ReplicaNotFoundException.NON_RBW_REPLICA)) {\n    rebuildPipelineWithFreshLocations(); // often another recovery already finalized it\n  } else { throw rnfe; }\n}","preventionTips":["Always fetch fresh LocatedBlocks before rebuilding a pipeline after any failure","Expect transient NON_RBW during racing recoveries: retry instead of failing the job","Avoid initiating recovery against DNs that just restarted until they re-registered"],"tags":["hdfs","datanode","block-recovery","replica-state","rbw"],"backgroundTag":"hdfs-replica-state-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}