apache/hadoop · warning · RecoveryInProgressException

rur.getRecoveryID() >= recoveryId = {recoveryId}, block={blo

Error message

rur.getRecoveryID() >= recoveryId = {recoveryId}, block={block}, rur={rur}

What it means

RecoveryInProgressException thrown by FsDatasetImpl.initReplicaRecoveryImpl when the replica is already RUR (Under Recovery) and its stored recoveryID is >= the incoming recoveryId. It means a concurrent recovery session with the same or newer id owns the replica; a second coordinator using an equal/older id is rejected so it can back off and retry with a higher id. This is a designed concurrency control exception, not corruption.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3117

    if (replica.getGenerationStamp() < block.getGenerationStamp()) {
      throw new IOException(
          "replica.getGenerationStamp() < block.getGenerationStamp(), block="
          + block + ", replica=" + replica);
    }

    //check recovery id
    if (replica.getGenerationStamp() >= recoveryId) {
      throw new IOException("THIS IS NOT SUPPOSED TO HAPPEN:"
          + " replica.getGenerationStamp() >= recoveryId = " + recoveryId
          + ", block=" + block + ", replica=" + replica);
    }

    //check RUR
    final ReplicaInfo rur;
    if (replica.getState() == ReplicaState.RUR) {
      rur = replica;
      if (rur.getRecoveryID() >= recoveryId) {
        throw new RecoveryInProgressException(
            "rur.getRecoveryID() >= recoveryId = " + recoveryId
            + ", block=" + block + ", rur=" + rur);
      }
      final long oldRecoveryID = rur.getRecoveryID();
      rur.setRecoveryID(recoveryId);
      LOG.info("initReplicaRecovery: update recovery id for " + block
          + " from " + oldRecoveryID + " to " + recoveryId);
    }
    else {
      rur = new ReplicaBuilder(ReplicaState.RUR)
          .from(replica).setRecoveryId(recoveryId).build();
      map.add(bpid, rur);
      LOG.info("initReplicaRecovery: changing replica state for "
          + block + " from " + replica.getState()
          + " to " + rur.getState());
      if (replica.getState() == ReplicaState.TEMPORARY || replica
          .getState() == ReplicaState.RBW) {
        ((ReplicaInPipeline) replica).releaseAllBytesReserved();

View on GitHub (pinned to 2add963021)

Solutions

1. Back off and retry: the next recovery attempt gets a higher recoveryId and succeeds once the first session commits.
2. As a client, prefer dfs.recoverLease(path) and poll its return instead of hammering immediate retries.
3. Ensure only one process coordinates recovery per file (avoid parallel recoverLease loops from monitoring scripts).
4. No DataNode-side change is needed - this exception is the protocol working as intended.

Example fix

// before: immediate retry loop
try { dataset.initReplicaRecovery(rBlock); }
catch (IOException e) { dataset.initReplicaRecovery(rBlock); }

// after: distinguish the retryable case and back off
try { dataset.initReplicaRecovery(rBlock); }
catch (RecoveryInProgressException e) {
  Thread.sleep(backoffMs);        // concurrent session owns the replica
  dataset.initReplicaRecovery(rBlock); // next id is higher; succeeds
}

boolean isRecoveryInProgress(IOException e) {
  return e instanceof org.apache.hadoop.hdfs.protocol.RecoveryInProgressException;
}

try {
  dataset.initReplicaRecovery(rBlock);
} catch (RecoveryInProgressException e) {
  // another coordinator with equal/newer id owns the replica: exponential backoff, then retry
  backoffAndRetry(rBlock, initialDelayMs, maxAttempts);
}

When it happens

Trigger: Two lease-recovery attempts overlap: the first initReplicaRecovery set rur.setRecoveryID(recoveryId1); a second request arrives with recoveryId2 <= recoveryId1 and hits rur.getRecoveryID() >= recoveryId.

Common situations: Client-triggered recoverLease racing the NameNode's own lease recovery; two clients recovering the same file after lease expiry; retry of a recovery RPC after network delay so the original already registered.

Related errors

• Replica was found but missing fields.
• {} has no enough internal blocks(current: {}), unable to sta
• Replica gen stamp < block genstamp, block={block}, replica={
• Interrupted while running disk check
• ProvidedReplica does not yet support writes

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a6be08ee09938a73. Report an issue: GitHub.