{"record":{"id":"013a7a4467c99459","repo":"apache/hadoop","slug":"cannot-recover-replica","errorCode":null,"errorMessage":"Cannot recover replica: {}","messagePattern":"Cannot recover replica: (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaUnderRecovery.java","lineNumber":42,"sourceCode":"/**\n * This class represents replicas that are under block recovery\n * It has a recovery id that is equal to the generation stamp \n * that the replica will be bumped to after recovery\n * The recovery id is used to handle multiple concurrent block recoveries.\n * A recovery with higher recovery id preempts recoveries with a lower id.\n *\n */\npublic class ReplicaUnderRecovery extends LocalReplica {\n  private LocalReplica original; // original replica to be recovered\n  private long recoveryId; // recovery id; it is also the generation stamp \n                           // that the replica will be bumped to after recovery\n\n  public ReplicaUnderRecovery(ReplicaInfo replica, long recoveryId) {\n    super(replica, replica.getVolume(), ((LocalReplica)replica).getDir());\n    if ( replica.getState() != ReplicaState.FINALIZED &&\n         replica.getState() != ReplicaState.RBW &&\n         replica.getState() != ReplicaState.RWR ) {\n      throw new IllegalArgumentException(\"Cannot recover replica: \" + replica);\n    }\n    this.original = (LocalReplica) replica;\n    this.recoveryId = recoveryId;\n  }\n\n  /**\n   * Copy constructor.\n   * @param from where to copy from\n   */\n  public ReplicaUnderRecovery(ReplicaUnderRecovery from) {\n    super(from);\n    this.original = (LocalReplica) from.getOriginalReplica();\n    this.recoveryId = from.getRecoveryID();\n  }\n\n  @Override\n  public long getRecoveryID() {\n    return recoveryId;","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaUnderRecovery.java#L24-L60","documentation":"ReplicaUnderRecovery's constructor only accepts replicas in FINALIZED, RBW or RWR state - the three states from which a block can legitimately be recovered (lease/block recovery). Wrapping anything else (e.g., a TEMPORARY replica, or an already-wrapped ReplicaUnderRecovery whose state is RUR) throws IllegalArgumentException with the replica's toString().","triggerScenarios":"new ReplicaUnderRecovery(replica, recoveryId) where replica.getState() is TEMPORARY or RUR (or any non-finalized/RBW/RWR value). Reached from DataNode.initBlockRecovery() during block recovery when the replica found in the volume map is a temporary file or is already under recovery from a previous attempt that was not cleaned up.","commonSituations":"A crashed/repeated block recovery leaving a stale RUR replica in the volume map that a new recovery attempt tries to wrap again Recovering a block whose only local copy is a temporary replica being written Inconsistent FsDataset state after a DataNode restart mid-recovery or after disk corruption","solutions":["Recover the ORIGINAL replica: call getOriginalReplica() on the existing ReplicaUnderRecovery and wrap that instead of double-wrapping","If the stale RUR is leftover garbage, remove/replace it in the volume map (restart the DataNode so replica state is re-scanned from disk)","Check replica.getState() against FINALIZED/RBW/RWR before constructing ReplicaUnderRecovery"],"exampleFix":"// before\nReplicaUnderRecovery rur =\n    new ReplicaUnderRecovery(volumeMap.get(b), recoveryId);\n// throws if that entry is already a ReplicaUnderRecovery (RUR)\n\n// after\nReplicaInfo cur = volumeMap.get(b);\nif (cur instanceof ReplicaUnderRecovery) {\n  cur = ((ReplicaUnderRecovery) cur).getOriginalReplica();\n}\nReplicaUnderRecovery rur = new ReplicaUnderRecovery(cur, recoveryId);","handlingStrategy":"type-guard","validationCode":"static boolean isRecoverable(ReplicaInfo r) {\n  ReplicaState s = r.getState();\n  return s == ReplicaState.FINALIZED\n      || s == ReplicaState.RBW\n      || s == ReplicaState.RWR;\n}","typeGuard":"static ReplicaInfo unwrapForRecovery(ReplicaInfo r) {\n  return (r instanceof ReplicaUnderRecovery)\n      ? ((ReplicaUnderRecovery) r).getOriginalReplica()\n      : r;\n}\n// then: new ReplicaUnderRecovery(unwrapForRecovery(r), recoveryId)","tryCatchPattern":"try { new ReplicaUnderRecovery(replica, recoveryId); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith(\"Cannot recover replica\")) { replica = ((ReplicaUnderRecovery) replica).getOriginalReplica(); } else throw e; }","preventionTips":["Always unwrap ReplicaUnderRecovery to its original before re-wrapping in repeated recovery","Guard recovery entry points with a FINALIZED/RBW/RWR state check"],"tags":["hdfs","datanode","block-recovery","replica","state-machine","illegal-argument"],"backgroundTag":"invalid-state-transition","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}