apache/hadoop · error · IOException

replica.getState() != RUR, replica={replica}

Error message

replica.getState() != RUR, replica={replica}

What it means

Thrown from FsDatasetImpl.updateReplica, the DataNode step that finalizes a replica during HDFS block recovery (lease, append, or truncate recovery). Recovery starts by converting the replica to ReplicaState.RUR via initReplicaRecovery; this check requires that conversion to have happened for the same attempt. If the volume map returns the replica as FINALIZED, RBW, or RWR, the recovery handshake is out of order, duplicated, or was reset between RPCs.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3165

    long startTimeMs = Time.monotonicNow();
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME,
        oldBlock.getBlockPoolId(), getStorageUuidForLock(oldBlock))) {
      //get replica
      final String bpid = oldBlock.getBlockPoolId();
      final ReplicaInfo replica = volumeMap.get(bpid, oldBlock.getBlockId());
      LOG.info("updateReplica: " + oldBlock
          + ", recoveryId=" + recoveryId
          + ", length=" + newlength
          + ", replica=" + replica);

      //check replica
      if (replica == null) {
        throw new ReplicaNotFoundException(oldBlock);
      }

      //check replica state
      if (replica.getState() != ReplicaState.RUR) {
        throw new IOException("replica.getState() != " + ReplicaState.RUR
            + ", replica=" + replica);
      }

      //check replica's byte on disk
      if (replica.getBytesOnDisk() != oldBlock.getNumBytes()) {
        throw new IOException("THIS IS NOT SUPPOSED TO HAPPEN:"
            + " replica.getBytesOnDisk() != block.getNumBytes(), block="
            + oldBlock + ", replica=" + replica);
      }

      //check replica files before update
      checkReplicaFiles(replica);

      //update replica
      final ReplicaInfo finalized = updateReplicaUnderRecovery(oldBlock
          .getBlockPoolId(), replica, recoveryId,
          newBlockId, newlength);

View on GitHub (pinned to 2add963021)

Solutions

1. Correlate preceding datanode log lines: the 'updateReplica: ... recoveryId=' entry and the earlier initReplicaRecovery line show which attempt is stale; a duplicate recovery RPC is the usual cause.
2. Do not run recoverLease concurrently with appends on the same file; issue one recoverLease and wait for the NameNode commitBlockSynchronization before writing again.
3. In most cases the next recovery attempt with the higher generation stamp re-initializes the RUR and self-heals; verify the file closes with 'hdfs fsck'.
4. If it persists on one DataNode, check for generation-stamp anomalies on the NameNode and upgrade to the latest point release of your branch.
5. As a last resort restart the DataNode so the volume map is rebuilt from disk, clearing stale RUR state.

Replica r = fsDataset.getReplica(bpid, block.getBlockId());
if (r == null || r.getState() != ReplicaState.RUR) {
  // recovery not initialized for this attempt; call initReplicaRecovery first
  return;
}

static boolean isRur(Replica r) {
  return r != null && r.getState() == ReplicaState.RUR;
}

try {
  fsDataset.updateReplica(block, recoveryId, newlength);
} catch (IOException e) {
  // report failure to the NameNode; the next recovery with a higher GS re-inits the RUR
  LOG.warn("updateReplica failed for " + block, e);
}

When it happens

Trigger: A retried or second block-recovery command reaches updateReplica after the previous recovery already finalized the RUR; the DataNode restarted between initReplicaRecovery and updateReplica so the volume map holds a fresh FINALIZED/RBW replica; an append or truncate raced the lease recovery and re-created the replica; the NameNode resends recovery with a stale generation stamp.

Common situations: Repeated recoverLease calls by HBase/MapReduce clients after append timeouts; readers forcing lease recovery while a writer retries; NameNode failover re-issuing pending recovery requests; version skew where recovery-id handling changed.

Related errors

• Replica was found but missing fields.
• Cannot append to an unfinalized replica {b}
• Cannot recover append/close to a replica that's not FINALIZE
• Cannot append to a replica with unexpected generation stamp
• Cannot recover a non-RBW replica {replicaInfo}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9f7d53f858ce1961. Report an issue: GitHub.