apache/hadoop · warning · RecoveryInProgressException

Failed to {} {} for {} on {} because another recovery is in

Error message

Failed to {} {} for {} on {} because another recovery is in progress by {} on {}

What it means

The file's last block is in BlockUCState.UNDER_RECOVERY — another recovery (initiated by clientName on uc.getClientMachine()) already owns it — so a concurrent recovery or open-for-write request is rejected with RecoveryInProgressException naming the competing client and machine.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:3024

        //
        // If the original holder has not renewed in the last SOFTLIMIT 
        // period, then start lease recovery.
        //
        if (lease.expiredSoftLimit()) {
          LOG.info("startFile: recover " + lease + ", src=" + src + " client "
              + clientName);
          if (internalReleaseLease(lease, src, iip, null)) {
            return true;
          } else {
            throw new RecoveryInProgressException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "lease recovery is in progress. Try again later."));
          }
        } else {
          final BlockInfo lastBlock = file.getLastBlock();
          if (lastBlock != null
              && lastBlock.getBlockUCState() == BlockUCState.UNDER_RECOVERY) {
            throw new RecoveryInProgressException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "another recovery is in progress by "
                        + clientName + " on " + uc.getClientMachine()));
          } else {
            throw new AlreadyBeingCreatedException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "this file lease is currently owned by "
                        + clientName + " on " + uc.getClientMachine()));
          }
        }
      }
    } else {
      return true;
     }
  }

  /**
   * Append to an existing file in the namespace.

View on GitHub (pinned to 2add963021)

Solutions

1. Retry after a short backoff — the first recovery wins and completes
2. Ensure exactly one actor triggers recovery (one monitor, not per-task auto-recovery)
3. If stuck, check the DataNodes hosting the last block; recovery needs a live replica

Example fix

// before
dfs.recoverLease(path);   // RecoveryInProgressException: another recovery in progress
// after
long deadline = System.currentTimeMillis() + 60_000L;
while (!dfs.isFileClosed(path) && System.currentTimeMillis() < deadline) {
  try { dfs.recoverLease(path); } catch (RecoveryInProgressException e) { /* another winner, wait */ }
  Thread.sleep(3000L);
}

catch (RecoveryInProgressException e) { if (e.getMessage() != null && e.getMessage().contains("another recovery is in progress")) { sleep(backoff); retry(); } else throw e; }

When it happens

Trigger: Two actors trigger recovery of the same under-construction file at once: one host's append/recoverLease while the NameNode lease monitor or another host's recoverLease already put the last block into UNDER_RECOVERY.

Common situations: Retry storm after a writer crash (several tasks/monitors call recoverLease concurrently); HA failover recovery plus eager client retry; monitoring daemons that auto-recover leases on live files.

Related errors

• Commit or complete block {commitBlock}, whereas it is under
• Failed to TRUNCATE_FILE <src> for <clientName> on <clientMac
• Failed to {} {} for {} on {} because lease recovery is in pr
• Replica was found but missing fields.
• Another {name} is running.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/948316654f05fcd6. Report an issue: GitHub.