apache/hadoop · warning · RecoveryInProgressException

Failed to {} {} for {} on {} because lease recovery is in pr

Error message

Failed to {} {} for {} on {} because lease recovery is in progress. Try again later.

What it means

The file's lease passed its soft limit, so the NameNode called internalReleaseLease to close or hand over the file; when block recovery cannot complete synchronously it returns false and the requesting client receives RecoveryInProgressException('lease recovery is in progress. Try again later.') — a transient come-back-later answer, not a permanent failure.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:3016

        // close only the file src
        LOG.info("recoverLease: " + lease + ", src=" + src +
          " from client " + clientName);
        return internalReleaseLease(lease, src, iip, holder);
      } else {
        assert lease.getHolder().equals(clientName) :
          "Current lease holder " + lease.getHolder() +
          " does not match file creator " + clientName;
        //
        // If the original holder has not renewed in the last SOFTLIMIT 
        // period, then start lease recovery.
        //
        if (lease.expiredSoftLimit()) {
          LOG.info("startFile: recover " + lease + ", src=" + src + " client "
              + clientName);
          if (internalReleaseLease(lease, src, iip, null)) {
            return true;
          } else {
            throw new RecoveryInProgressException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "lease recovery is in progress. Try again later."));
          }
        } else {
          final BlockInfo lastBlock = file.getLastBlock();
          if (lastBlock != null
              && lastBlock.getBlockUCState() == BlockUCState.UNDER_RECOVERY) {
            throw new RecoveryInProgressException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "another recovery is in progress by "
                        + clientName + " on " + uc.getClientMachine()));
          } else {
            throw new AlreadyBeingCreatedException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "this file lease is currently owned by "
                        + clientName + " on " + uc.getClientMachine()));
          }
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry with backoff — recovery usually completes within seconds once DataNodes respond
2. If it persists, check DataNode liveness for the last block's replicas (hdfs fsck <file>); recovery cannot finish without reachable DataNodes
3. If the file is disposable, delete it instead of waiting

Example fix

// before
FSDataOutputStream out = fs.append(path);   // RecoveryInProgressException
// after
for (int i = 0; i < 10; i++) {
  try { out = fs.append(path); break; }
  catch (RecoveryInProgressException e) { Thread.sleep(2000L * (i + 1)); }
}

catch (RecoveryInProgressException e) { sleep(backoff); retryAppend(); } // lease recovery finishes asynchronously once DataNodes respond

When it happens

Trigger: create/append/recoverLease racing an in-flight lease recovery: the previous writer stalled past the soft limit (60s), the NN initiated recovery, and blocks are still being finalized with DataNodes when a new client requests the file.

Common situations: Failed writer task followed immediately by a retry; recovery blocked on a slow or dead DataNode holding the last block; HA failover triggering recoveries that client retries then collide with.

Related errors

• {}. Name node is in safe mode. {} NamenodeHostName:{}
• Failed to {} {} for {} on {} because another recovery is in
• Replica was found but missing fields.
• Trying to commit inconsistent block: id = {blockId}, expecte
• Commit block with mismatching GS. NN has {block}, client sub

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/516b8be38d19da96. Report an issue: GitHub.