{"record":{"id":"9f7d53f858ce1961","repo":"apache/hadoop","slug":"replica-getstate-rur-replica-replica","errorCode":null,"errorMessage":"replica.getState() != RUR, replica={replica}","messagePattern":"replica\\.getState\\(\\) != RUR, replica=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3165,"sourceCode":"    long startTimeMs = Time.monotonicNow();\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME,\n        oldBlock.getBlockPoolId(), getStorageUuidForLock(oldBlock))) {\n      //get replica\n      final String bpid = oldBlock.getBlockPoolId();\n      final ReplicaInfo replica = volumeMap.get(bpid, oldBlock.getBlockId());\n      LOG.info(\"updateReplica: \" + oldBlock\n          + \", recoveryId=\" + recoveryId\n          + \", length=\" + newlength\n          + \", replica=\" + replica);\n\n      //check replica\n      if (replica == null) {\n        throw new ReplicaNotFoundException(oldBlock);\n      }\n\n      //check replica state\n      if (replica.getState() != ReplicaState.RUR) {\n        throw new IOException(\"replica.getState() != \" + ReplicaState.RUR\n            + \", replica=\" + replica);\n      }\n\n      //check replica's byte on disk\n      if (replica.getBytesOnDisk() != oldBlock.getNumBytes()) {\n        throw new IOException(\"THIS IS NOT SUPPOSED TO HAPPEN:\"\n            + \" replica.getBytesOnDisk() != block.getNumBytes(), block=\"\n            + oldBlock + \", replica=\" + replica);\n      }\n\n      //check replica files before update\n      checkReplicaFiles(replica);\n\n      //update replica\n      final ReplicaInfo finalized = updateReplicaUnderRecovery(oldBlock\n          .getBlockPoolId(), replica, recoveryId,\n          newBlockId, newlength);\n","sourceCodeStart":3147,"sourceCodeEnd":3183,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3147-L3183","documentation":"Thrown from FsDatasetImpl.updateReplica, the DataNode step that finalizes a replica during HDFS block recovery (lease, append, or truncate recovery). Recovery starts by converting the replica to ReplicaState.RUR via initReplicaRecovery; this check requires that conversion to have happened for the same attempt. If the volume map returns the replica as FINALIZED, RBW, or RWR, the recovery handshake is out of order, duplicated, or was reset between RPCs.","triggerScenarios":"A retried or second block-recovery command reaches updateReplica after the previous recovery already finalized the RUR; the DataNode restarted between initReplicaRecovery and updateReplica so the volume map holds a fresh FINALIZED/RBW replica; an append or truncate raced the lease recovery and re-created the replica; the NameNode resends recovery with a stale generation stamp.","commonSituations":"Repeated recoverLease calls by HBase/MapReduce clients after append timeouts; readers forcing lease recovery while a writer retries; NameNode failover re-issuing pending recovery requests; version skew where recovery-id handling changed.","solutions":["Correlate preceding datanode log lines: the 'updateReplica: ... recoveryId=' entry and the earlier initReplicaRecovery line show which attempt is stale; a duplicate recovery RPC is the usual cause.","Do not run recoverLease concurrently with appends on the same file; issue one recoverLease and wait for the NameNode commitBlockSynchronization before writing again.","In most cases the next recovery attempt with the higher generation stamp re-initializes the RUR and self-heals; verify the file closes with 'hdfs fsck'.","If it persists on one DataNode, check for generation-stamp anomalies on the NameNode and upgrade to the latest point release of your branch.","As a last resort restart the DataNode so the volume map is rebuilt from disk, clearing stale RUR state."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Replica r = fsDataset.getReplica(bpid, block.getBlockId());\nif (r == null || r.getState() != ReplicaState.RUR) {\n  // recovery not initialized for this attempt; call initReplicaRecovery first\n  return;\n}","typeGuard":"static boolean isRur(Replica r) {\n  return r != null && r.getState() == ReplicaState.RUR;\n}","tryCatchPattern":"try {\n  fsDataset.updateReplica(block, recoveryId, newlength);\n} catch (IOException e) {\n  // report failure to the NameNode; the next recovery with a higher GS re-inits the RUR\n  LOG.warn(\"updateReplica failed for \" + block, e);\n}","preventionTips":["Serialize lease recovery: one recoverLease call, wait for commitBlockSynchronization before appending again.","Keep a single writer per file; concurrent writers plus recoverLease is the main producer of stale RUR state.","Monitor DN logs for repeated recovery of the same block id - it indicates a client retry loop."],"tags":["hdfs","datanode","block-recovery","replica-state","lease-recovery"],"backgroundTag":"hdfs-block-recovery-state-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}