{"record":{"id":"3d8caf1ffb0cad07","repo":"apache/hadoop","slug":"dir-namesystem-internalreleaselease-failed-to-re","errorCode":null,"errorMessage":"DIR* NameSystem.internalReleaseLease: Failed to release lease for file {}. Committed blocks are waiting to be minimally replicated.","messagePattern":"DIR\\* NameSystem\\.internalReleaseLease: Failed to release lease for file (.+?)\\. Committed blocks are waiting to be minimally replicated\\.","errorType":"exception","errorClass":"AlreadyBeingCreatedException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java","lineNumber":3854,"sourceCode":"      if(penultimateBlockMinStorage &&\n          blockManager.hasMinStorage(lastBlock)) {\n        finalizeINodeFileUnderConstruction(src, pendingFile,\n            iip.getLatestSnapshotId(), false);\n        NameNode.stateChangeLog.warn(\"BLOCK*\" +\n            \" internalReleaseLease: Committed blocks are minimally\" +\n            \" replicated, lease removed, file\" + src + \" closed.\");\n        return true;  // closed!\n      }\n      // Cannot close file right now, since some blocks \n      // are not yet minimally replicated.\n      // This may potentially cause infinite loop in lease recovery\n      // if there are no valid replicas on data-nodes.\n      String message = \"DIR* NameSystem.internalReleaseLease: \" +\n          \"Failed to release lease for file \" + src +\n          \". Committed blocks are waiting to be minimally replicated.\";\n      NameNode.stateChangeLog.warn(message);\n      if (!penultimateBlockMinStorage) {\n        throw new AlreadyBeingCreatedException(message);\n      }\n      // Intentionally fall through to UNDER_RECOVERY so BLOCK_RECOVERY is\n      // attempted\n    case UNDER_CONSTRUCTION:\n    case UNDER_RECOVERY:\n      BlockUnderConstructionFeature uc =\n          lastBlock.getUnderConstructionFeature();\n      // determine if last block was intended to be truncated\n      BlockInfo recoveryBlock = uc.getTruncateBlock();\n      boolean truncateRecovery = recoveryBlock != null;\n      boolean copyOnTruncate = truncateRecovery &&\n          recoveryBlock.getBlockId() != lastBlock.getBlockId();\n      assert !copyOnTruncate ||\n          recoveryBlock.getBlockId() < lastBlock.getBlockId() &&\n          recoveryBlock.getGenerationStamp() < lastBlock.getGenerationStamp() &&\n          recoveryBlock.getNumBytes() > lastBlock.getNumBytes() :\n            \"wrong recoveryBlock\";\n","sourceCodeStart":3836,"sourceCodeEnd":3872,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L3836-L3872","documentation":"internalReleaseLease cannot close the file because the committed last block has not reached minimal replication and the penultimate block is also below min storage. It throws AlreadyBeingCreatedException so lease recovery is retried later; the log explicitly warns that recovery can loop forever if no valid replicas exist on the datanodes.","triggerScenarios":"Lease recovery while the last (COMMITTED) block and/or penultimate block are under-replicated: pipeline datanodes died before commit, replicas lost to decommissioning or disk failures.","commonSituations":"Writer crashed and its pipeline DNs went down in the same incident; small clusters where one dead DN drops replication below minReplication; flaky disks losing replica files.","solutions":["Restore the datanodes that hold replicas (restart them, fix mounts) so replication can be met, then retry recovery","Use 'hdfs debug recoverLease -path <path> -retries N' to drive repeated attempts","Check 'hdfs dfsadmin -report' and under-replicated block count; fix under-replication first","If replicas are permanently lost, salvage readable bytes (dfs -get), delete the file, and re-ingest"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  dfs.recoverLease(path);\n} catch (AlreadyBeingCreatedException e) {\n  // blocks under-replicated: wait for datanodes/replication, then retry\n  waitForMinReplication(path, timeout);\n  dfs.recoverLease(path);\n}","preventionTips":["Keep datanode health good; dead pipeline DNs block lease recovery","Monitor under-replicated blocks and fix them before crashes force recovery","Use 'hdfs debug recoverLease -retries' to space out attempts"],"tags":["hdfs","namenode","lease-recovery","under-replicated","datanode"],"backgroundTag":"under-replicated-blocks","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}