apache/hadoop · error · IOException

rur.getRecoveryID() != recoveryId = {recoveryId}, rur={rur}

Error message

rur.getRecoveryID() != recoveryId = {recoveryId}, rur={rur}

What it means

FsDatasetImpl.updateReplicaUnderRecovery requires the ReplicaUnderRecovery on disk to carry the same recoveryId (the new generation stamp) the current attempt commits. initReplicaRecovery stamps this id when recovery starts, and a later higher id overwrites it. A mismatch means updateReplica arrived for a different recovery attempt than the one that initialized or last refreshed the RUR.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3222

      return finalized;
    } finally {
      if (dataNodeMetrics != null) {
        long updateReplicaUnderRecoveryMs = Time.monotonicNow() - startTimeMs;
        dataNodeMetrics.addUpdateReplicaUnderRecoveryOp(
            updateReplicaUnderRecoveryMs);
      }
    }
  }

  private ReplicaInfo updateReplicaUnderRecovery(
                                          String bpid,
                                          ReplicaInfo rur,
                                          long recoveryId,
                                          long newBlockId,
                                          long newlength) throws IOException {
    //check recovery id
    if (rur.getRecoveryID() != recoveryId) {
      throw new IOException("rur.getRecoveryID() != recoveryId = " + recoveryId
          + ", rur=" + rur);
    }

    boolean copyOnTruncate = newBlockId > 0L && rur.getBlockId() != newBlockId;
    // bump rur's GS to be recovery id
    if(!copyOnTruncate) {
      rur.bumpReplicaGS(recoveryId);
    }

    //update length
    if (rur.getNumBytes() < newlength) {
      throw new IOException("rur.getNumBytes() < newlength = " + newlength
          + ", rur=" + rur);
    }

    if (rur.getNumBytes() > newlength) {
      if(!copyOnTruncate) {
        rur.breakHardLinksIfNeeded();

View on GitHub (pinned to 2add963021)

Solutions

1. Compare the 'recoveryId=' value in the exception with the 'Recovering ... new GS' line from initReplicaRecovery in the DN log; the delta identifies the stale attempt.
2. Stop concurrent recoverLease loops; a single recovery with the highest GS re-initializes the RUR (initReplicaRecovery bumps the stored recovery id) and the next attempt succeeds.
3. Verify the file eventually closes with 'hdfs fsck'; if it stays open, close the writer and force a fresh lease.
4. Upgrade if your release is affected by known recovery-id races; report with logs if both ids come from the same NN session.

Replica r = fsDataset.getReplica(bpid, block.getBlockId());
if (r instanceof ReplicaUnderRecovery
    && ((ReplicaUnderRecovery) r).getRecoveryID() != recoveryId) {
  // re-run initReplicaRecovery with the new id so the RUR is re-stamped
}

static boolean matchesRecoveryId(Replica r, long recoveryId) {
  return r instanceof ReplicaUnderRecovery
      && ((ReplicaUnderRecovery) r).getRecoveryID() == recoveryId;
}

try {
  fsDataset.updateReplica(block, recoveryId, newlength);
} catch (IOException e) {
  if (e.getMessage().contains("getRecoveryID()")) {
    // stale recovery attempt: drop it and let the highest-id recovery win
  }
}

When it happens

Trigger: Two interleaved recoveries: recovery A initialized the RUR, then recovery B (higher generation stamp) calls updateReplica before re-initializing the RUR; the NameNode retried the recovery RPC with a different stamp; append or truncate on the block raced the recovery.

Common situations: Multiple clients calling recoverLease in a loop; NameNode failover re-issuing recovery; HBase lease-recovery storms; mixed NN/DN versions with different recovery-id semantics.

Related errors

• Cannot append to a replica with unexpected generation stamp
• Replica was found but missing fields.
• Replica gen stamp < block genstamp, block={block}, replica={
• ProvidedReplica does not yet support writes
• The new recovery id: {} must be greater than the current one

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8420be361cc52bff. Report an issue: GitHub.