apache/hadoop · error · IllegalArgumentException

The new recovery id: {} must be greater than the current one

Error message

The new recovery id: {} must be greater than the current one: {}

What it means

ReplicaUnderRecovery.setRecoveryID(long) enforces strictly increasing recovery ids: the recoveryId doubles as the generation stamp the replica will be bumped to after recovery, so a new recovery attempt must carry a larger id than the current one. Passing an id less than or equal to the current one throws IllegalArgumentException showing both values.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaUnderRecovery.java:68

   * @param from where to copy from
   */
  public ReplicaUnderRecovery(ReplicaUnderRecovery from) {
    super(from);
    this.original = (LocalReplica) from.getOriginalReplica();
    this.recoveryId = from.getRecoveryID();
  }

  @Override
  public long getRecoveryID() {
    return recoveryId;
  }

  @Override
  public void setRecoveryID(long recoveryId) {
    if (recoveryId > this.recoveryId) {
      this.recoveryId = recoveryId;
    } else {
      throw new IllegalArgumentException("The new recovery id: " + recoveryId
          + " must be greater than the current one: " + this.recoveryId);
    }
  }

  /**
   * Get the original replica that's under recovery
   * @return the original replica under recovery
   */
  @Override
  public ReplicaInfo getOriginalReplica() {
    return original;
  }
  
  @Override //ReplicaInfo
  public ReplicaState getState() {
    return ReplicaState.RUR;
  }
  

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure each new recovery attempt uses a recoveryId strictly greater than the previous (normally the new requested block generation stamp)
2. Treat this exception as a signal the recovery request is stale: log and skip rather than crash
3. For NameNode HA: verify the active NN is the one issuing recovery so stale standbys cannot replay old ids

Example fix

// before
rur.setRecoveryID(newRecoveryId); // throws if newRecoveryId <= rur.getRecoveryID()

// after
if (newRecoveryId > rur.getRecoveryID()) {
  rur.setRecoveryID(newRecoveryId);
} else {
  LOG.warn("Stale recovery id {} for {} - keeping {}",
      newRecoveryId, rur, rur.getRecoveryID());
}

void bumpRecoveryId(ReplicaUnderRecovery r, long newId) {
  if (newId > r.getRecoveryID()) {
    r.setRecoveryID(newId);
  } else {
    LOG.debug("Ignoring stale recovery id {} <= {}", newId, r.getRecoveryID());
  }
}

try { rur.setRecoveryID(newId); } catch (IllegalArgumentException e) { /* stale recovery request: drop it, do not crash the DN thread */ LOG.warn("Stale recovery request: {}", e.getMessage()); }

When it happens

Trigger: DataNode.updateReplicaUnderRecovery()/sync blocks call setRecoveryID(newId) when a later recovery attempt (higher requested block GS) arrives for a replica already under recovery, and newId <= the stored recoveryId. Happens with a stale NameNode/primary retried recovery request, an old client resending a recovery command, or clock/sequence regressions in test-generated recovery ids.

Common situations: Replayed or out-of-order block-recovery RPCs from an HA NameNode that lagged behind (standby resending an old recovery task); tests that reuse the same recoveryId across iterations; a retry of the same recovery attempt instead of a fresh, higher one.

Related errors

• Cannot recover replica: {}
• Replica gen stamp < block genstamp, block={block}, replica={
• ProvidedReplica does not yet support writes
• Missing a valid replica to recover from
• Invalid state for recovering from replica with blk id {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6a56be573ce79a5f. Report an issue: GitHub.