{"record":{"id":"8420be361cc52bff","repo":"apache/hadoop","slug":"rur-getrecoveryid-recoveryid-recoveryid-8420be","errorCode":null,"errorMessage":"rur.getRecoveryID() != recoveryId = {recoveryId}, rur={rur}","messagePattern":"rur\\.getRecoveryID\\(\\) != recoveryId = (.+?), rur=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3222,"sourceCode":"      return finalized;\n    } finally {\n      if (dataNodeMetrics != null) {\n        long updateReplicaUnderRecoveryMs = Time.monotonicNow() - startTimeMs;\n        dataNodeMetrics.addUpdateReplicaUnderRecoveryOp(\n            updateReplicaUnderRecoveryMs);\n      }\n    }\n  }\n\n  private ReplicaInfo updateReplicaUnderRecovery(\n                                          String bpid,\n                                          ReplicaInfo rur,\n                                          long recoveryId,\n                                          long newBlockId,\n                                          long newlength) throws IOException {\n    //check recovery id\n    if (rur.getRecoveryID() != recoveryId) {\n      throw new IOException(\"rur.getRecoveryID() != recoveryId = \" + recoveryId\n          + \", rur=\" + rur);\n    }\n\n    boolean copyOnTruncate = newBlockId > 0L && rur.getBlockId() != newBlockId;\n    // bump rur's GS to be recovery id\n    if(!copyOnTruncate) {\n      rur.bumpReplicaGS(recoveryId);\n    }\n\n    //update length\n    if (rur.getNumBytes() < newlength) {\n      throw new IOException(\"rur.getNumBytes() < newlength = \" + newlength\n          + \", rur=\" + rur);\n    }\n\n    if (rur.getNumBytes() > newlength) {\n      if(!copyOnTruncate) {\n        rur.breakHardLinksIfNeeded();","sourceCodeStart":3204,"sourceCodeEnd":3240,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3204-L3240","documentation":"FsDatasetImpl.updateReplicaUnderRecovery requires the ReplicaUnderRecovery on disk to carry the same recoveryId (the new generation stamp) the current attempt commits. initReplicaRecovery stamps this id when recovery starts, and a later higher id overwrites it. A mismatch means updateReplica arrived for a different recovery attempt than the one that initialized or last refreshed the RUR.","triggerScenarios":"Two interleaved recoveries: recovery A initialized the RUR, then recovery B (higher generation stamp) calls updateReplica before re-initializing the RUR; the NameNode retried the recovery RPC with a different stamp; append or truncate on the block raced the recovery.","commonSituations":"Multiple clients calling recoverLease in a loop; NameNode failover re-issuing recovery; HBase lease-recovery storms; mixed NN/DN versions with different recovery-id semantics.","solutions":["Compare the 'recoveryId=' value in the exception with the 'Recovering ... new GS' line from initReplicaRecovery in the DN log; the delta identifies the stale attempt.","Stop concurrent recoverLease loops; a single recovery with the highest GS re-initializes the RUR (initReplicaRecovery bumps the stored recovery id) and the next attempt succeeds.","Verify the file eventually closes with 'hdfs fsck'; if it stays open, close the writer and force a fresh lease.","Upgrade if your release is affected by known recovery-id races; report with logs if both ids come from the same NN session."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Replica r = fsDataset.getReplica(bpid, block.getBlockId());\nif (r instanceof ReplicaUnderRecovery\n    && ((ReplicaUnderRecovery) r).getRecoveryID() != recoveryId) {\n  // re-run initReplicaRecovery with the new id so the RUR is re-stamped\n}","typeGuard":"static boolean matchesRecoveryId(Replica r, long recoveryId) {\n  return r instanceof ReplicaUnderRecovery\n      && ((ReplicaUnderRecovery) r).getRecoveryID() == recoveryId;\n}","tryCatchPattern":"try {\n  fsDataset.updateReplica(block, recoveryId, newlength);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"getRecoveryID()\")) {\n    // stale recovery attempt: drop it and let the highest-id recovery win\n  }\n}","preventionTips":["Never run concurrent recoverLease loops against the same file.","Wait for the previous recovery to commit before starting a new one with a higher stamp.","On NN failover, let pending recoveries drain before re-issuing client recovery requests."],"tags":["hdfs","datanode","block-recovery","generation-stamp","lease-recovery"],"backgroundTag":"hdfs-block-recovery-state-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}