{"record":{"id":"516b8be38d19da96","repo":"apache/hadoop","slug":"failed-to-for-on-because-lease-recover","errorCode":null,"errorMessage":"Failed to {} {} for {} on {} because lease recovery is in progress. Try again later.","messagePattern":"Failed to (.+?) (.+?) for (.+?) on (.+?) because lease recovery is in progress\\. Try again later\\.","errorType":"exception","errorClass":"RecoveryInProgressException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java","lineNumber":3016,"sourceCode":"        // close only the file src\n        LOG.info(\"recoverLease: \" + lease + \", src=\" + src +\n          \" from client \" + clientName);\n        return internalReleaseLease(lease, src, iip, holder);\n      } else {\n        assert lease.getHolder().equals(clientName) :\n          \"Current lease holder \" + lease.getHolder() +\n          \" does not match file creator \" + clientName;\n        //\n        // If the original holder has not renewed in the last SOFTLIMIT \n        // period, then start lease recovery.\n        //\n        if (lease.expiredSoftLimit()) {\n          LOG.info(\"startFile: recover \" + lease + \", src=\" + src + \" client \"\n              + clientName);\n          if (internalReleaseLease(lease, src, iip, null)) {\n            return true;\n          } else {\n            throw new RecoveryInProgressException(\n                op.getExceptionMessage(src, holder, clientMachine,\n                    \"lease recovery is in progress. Try again later.\"));\n          }\n        } else {\n          final BlockInfo lastBlock = file.getLastBlock();\n          if (lastBlock != null\n              && lastBlock.getBlockUCState() == BlockUCState.UNDER_RECOVERY) {\n            throw new RecoveryInProgressException(\n                op.getExceptionMessage(src, holder, clientMachine,\n                    \"another recovery is in progress by \"\n                        + clientName + \" on \" + uc.getClientMachine()));\n          } else {\n            throw new AlreadyBeingCreatedException(\n                op.getExceptionMessage(src, holder, clientMachine,\n                    \"this file lease is currently owned by \"\n                        + clientName + \" on \" + uc.getClientMachine()));\n          }\n        }","sourceCodeStart":2998,"sourceCodeEnd":3034,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L2998-L3034","documentation":"The file's lease passed its soft limit, so the NameNode called internalReleaseLease to close or hand over the file; when block recovery cannot complete synchronously it returns false and the requesting client receives RecoveryInProgressException('lease recovery is in progress. Try again later.') — a transient come-back-later answer, not a permanent failure.","triggerScenarios":"create/append/recoverLease racing an in-flight lease recovery: the previous writer stalled past the soft limit (60s), the NN initiated recovery, and blocks are still being finalized with DataNodes when a new client requests the file.","commonSituations":"Failed writer task followed immediately by a retry; recovery blocked on a slow or dead DataNode holding the last block; HA failover triggering recoveries that client retries then collide with.","solutions":["Retry with backoff — recovery usually completes within seconds once DataNodes respond","If it persists, check DataNode liveness for the last block's replicas (hdfs fsck <file>); recovery cannot finish without reachable DataNodes","If the file is disposable, delete it instead of waiting"],"exampleFix":"// before\nFSDataOutputStream out = fs.append(path);   // RecoveryInProgressException\n// after\nfor (int i = 0; i < 10; i++) {\n  try { out = fs.append(path); break; }\n  catch (RecoveryInProgressException e) { Thread.sleep(2000L * (i + 1)); }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (RecoveryInProgressException e) { sleep(backoff); retryAppend(); } // lease recovery finishes asynchronously once DataNodes respond","preventionTips":["Treat RecoveryInProgressException as retryable by design, never as a fatal error","Keep DataNodes healthy so block recovery completes promptly"],"tags":["hdfs","lease-recovery","transient","retry"],"backgroundTag":"lease-recovery-in-progress","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}