apache/hadoop · error · IllegalArgumentException

Cannot recover replica: {}

Error message

Cannot recover replica: {}

What it means

ReplicaUnderRecovery's constructor only accepts replicas in FINALIZED, RBW or RWR state - the three states from which a block can legitimately be recovered (lease/block recovery). Wrapping anything else (e.g., a TEMPORARY replica, or an already-wrapped ReplicaUnderRecovery whose state is RUR) throws IllegalArgumentException with the replica's toString().

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaUnderRecovery.java:42

/**
 * This class represents replicas that are under block recovery
 * It has a recovery id that is equal to the generation stamp 
 * that the replica will be bumped to after recovery
 * The recovery id is used to handle multiple concurrent block recoveries.
 * A recovery with higher recovery id preempts recoveries with a lower id.
 *
 */
public class ReplicaUnderRecovery extends LocalReplica {
  private LocalReplica original; // original replica to be recovered
  private long recoveryId; // recovery id; it is also the generation stamp 
                           // that the replica will be bumped to after recovery

  public ReplicaUnderRecovery(ReplicaInfo replica, long recoveryId) {
    super(replica, replica.getVolume(), ((LocalReplica)replica).getDir());
    if ( replica.getState() != ReplicaState.FINALIZED &&
         replica.getState() != ReplicaState.RBW &&
         replica.getState() != ReplicaState.RWR ) {
      throw new IllegalArgumentException("Cannot recover replica: " + replica);
    }
    this.original = (LocalReplica) replica;
    this.recoveryId = recoveryId;
  }

  /**
   * Copy constructor.
   * @param from where to copy from
   */
  public ReplicaUnderRecovery(ReplicaUnderRecovery from) {
    super(from);
    this.original = (LocalReplica) from.getOriginalReplica();
    this.recoveryId = from.getRecoveryID();
  }

  @Override
  public long getRecoveryID() {
    return recoveryId;

View on GitHub (pinned to 2add963021)

Solutions

1. Recover the ORIGINAL replica: call getOriginalReplica() on the existing ReplicaUnderRecovery and wrap that instead of double-wrapping
2. If the stale RUR is leftover garbage, remove/replace it in the volume map (restart the DataNode so replica state is re-scanned from disk)
3. Check replica.getState() against FINALIZED/RBW/RWR before constructing ReplicaUnderRecovery

Example fix

// before
ReplicaUnderRecovery rur =
    new ReplicaUnderRecovery(volumeMap.get(b), recoveryId);
// throws if that entry is already a ReplicaUnderRecovery (RUR)

// after
ReplicaInfo cur = volumeMap.get(b);
if (cur instanceof ReplicaUnderRecovery) {
  cur = ((ReplicaUnderRecovery) cur).getOriginalReplica();
}
ReplicaUnderRecovery rur = new ReplicaUnderRecovery(cur, recoveryId);

static boolean isRecoverable(ReplicaInfo r) {
  ReplicaState s = r.getState();
  return s == ReplicaState.FINALIZED
      || s == ReplicaState.RBW
      || s == ReplicaState.RWR;
}

static ReplicaInfo unwrapForRecovery(ReplicaInfo r) {
  return (r instanceof ReplicaUnderRecovery)
      ? ((ReplicaUnderRecovery) r).getOriginalReplica()
      : r;
}
// then: new ReplicaUnderRecovery(unwrapForRecovery(r), recoveryId)

try { new ReplicaUnderRecovery(replica, recoveryId); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("Cannot recover replica")) { replica = ((ReplicaUnderRecovery) replica).getOriginalReplica(); } else throw e; }

When it happens

Trigger: new ReplicaUnderRecovery(replica, recoveryId) where replica.getState() is TEMPORARY or RUR (or any non-finalized/RBW/RWR value). Reached from DataNode.initBlockRecovery() during block recovery when the replica found in the volume map is a temporary file or is already under recovery from a previous attempt that was not cleaned up.

Common situations: A crashed/repeated block recovery leaving a stale RUR replica in the volume map that a new recovery attempt tries to wrap again Recovering a block whose only local copy is a temporary replica being written Inconsistent FsDataset state after a DataNode restart mid-recovery or after disk corruption

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Unknown replica state {} for PROVIDED replica
• The new recovery id: {} must be greater than the current one
• Missing a valid replica to recover from
• Invalid state for recovering from replica with blk id {}
• Trying to construct a provided replica on {} without enough

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/013a7a4467c99459. Report an issue: GitHub.