{"record":{"id":"a6be08ee09938a73","repo":"apache/hadoop","slug":"rur-getrecoveryid-recoveryid-recoveryid","errorCode":null,"errorMessage":"rur.getRecoveryID() >= recoveryId = {recoveryId}, block={block}, rur={rur}","messagePattern":"rur\\.getRecoveryID\\(\\) >= recoveryId = (.+?), block=(.+?), rur=(.+?)","errorType":"exception","errorClass":"RecoveryInProgressException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3117,"sourceCode":"    if (replica.getGenerationStamp() < block.getGenerationStamp()) {\n      throw new IOException(\n          \"replica.getGenerationStamp() < block.getGenerationStamp(), block=\"\n          + block + \", replica=\" + replica);\n    }\n\n    //check recovery id\n    if (replica.getGenerationStamp() >= recoveryId) {\n      throw new IOException(\"THIS IS NOT SUPPOSED TO HAPPEN:\"\n          + \" replica.getGenerationStamp() >= recoveryId = \" + recoveryId\n          + \", block=\" + block + \", replica=\" + replica);\n    }\n\n    //check RUR\n    final ReplicaInfo rur;\n    if (replica.getState() == ReplicaState.RUR) {\n      rur = replica;\n      if (rur.getRecoveryID() >= recoveryId) {\n        throw new RecoveryInProgressException(\n            \"rur.getRecoveryID() >= recoveryId = \" + recoveryId\n            + \", block=\" + block + \", rur=\" + rur);\n      }\n      final long oldRecoveryID = rur.getRecoveryID();\n      rur.setRecoveryID(recoveryId);\n      LOG.info(\"initReplicaRecovery: update recovery id for \" + block\n          + \" from \" + oldRecoveryID + \" to \" + recoveryId);\n    }\n    else {\n      rur = new ReplicaBuilder(ReplicaState.RUR)\n          .from(replica).setRecoveryId(recoveryId).build();\n      map.add(bpid, rur);\n      LOG.info(\"initReplicaRecovery: changing replica state for \"\n          + block + \" from \" + replica.getState()\n          + \" to \" + rur.getState());\n      if (replica.getState() == ReplicaState.TEMPORARY || replica\n          .getState() == ReplicaState.RBW) {\n        ((ReplicaInPipeline) replica).releaseAllBytesReserved();","sourceCodeStart":3099,"sourceCodeEnd":3135,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3099-L3135","documentation":"RecoveryInProgressException thrown by FsDatasetImpl.initReplicaRecoveryImpl when the replica is already RUR (Under Recovery) and its stored recoveryID is >= the incoming recoveryId. It means a concurrent recovery session with the same or newer id owns the replica; a second coordinator using an equal/older id is rejected so it can back off and retry with a higher id. This is a designed concurrency control exception, not corruption.","triggerScenarios":"Two lease-recovery attempts overlap: the first initReplicaRecovery set rur.setRecoveryID(recoveryId1); a second request arrives with recoveryId2 <= recoveryId1 and hits rur.getRecoveryID() >= recoveryId.","commonSituations":"Client-triggered recoverLease racing the NameNode's own lease recovery; two clients recovering the same file after lease expiry; retry of a recovery RPC after network delay so the original already registered.","solutions":["Back off and retry: the next recovery attempt gets a higher recoveryId and succeeds once the first session commits.","As a client, prefer dfs.recoverLease(path) and poll its return instead of hammering immediate retries.","Ensure only one process coordinates recovery per file (avoid parallel recoverLease loops from monitoring scripts).","No DataNode-side change is needed - this exception is the protocol working as intended."],"exampleFix":"// before: immediate retry loop\ntry { dataset.initReplicaRecovery(rBlock); }\ncatch (IOException e) { dataset.initReplicaRecovery(rBlock); }\n\n// after: distinguish the retryable case and back off\ntry { dataset.initReplicaRecovery(rBlock); }\ncatch (RecoveryInProgressException e) {\n  Thread.sleep(backoffMs);        // concurrent session owns the replica\n  dataset.initReplicaRecovery(rBlock); // next id is higher; succeeds\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"boolean isRecoveryInProgress(IOException e) {\n  return e instanceof org.apache.hadoop.hdfs.protocol.RecoveryInProgressException;\n}","tryCatchPattern":"try {\n  dataset.initReplicaRecovery(rBlock);\n} catch (RecoveryInProgressException e) {\n  // another coordinator with equal/newer id owns the replica: exponential backoff, then retry\n  backoffAndRetry(rBlock, initialDelayMs, maxAttempts);\n}","preventionTips":["Coordinate recovery through one path (DFSClient.recoverLease) instead of concurrent manual triggers.","Use exponential backoff on RecoveryInProgressException - retrying immediately just loses again.","Monitor how many concurrent lease recoveries your tooling fires per file."],"tags":["hdfs","datanode","block-recovery","recovery-in-progress","concurrency"],"backgroundTag":"recovery-in-progress","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}