{"record":{"id":"aa00b8220f978ccc","repo":"apache/hadoop","slug":"dir-namesystem-internalreleaselease-attempt-to-r","errorCode":null,"errorMessage":"DIR* NameSystem.internalReleaseLease: attempt to release a create lock on {} but file is already closed.","messagePattern":"DIR\\* NameSystem\\.internalReleaseLease: attempt to release a create lock on (.+?) but file is already closed\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java","lineNumber":3817,"sourceCode":"      finalizeINodeFileUnderConstruction(src, pendingFile,\n          iip.getLatestSnapshotId(), false);\n      NameNode.stateChangeLog.warn(\"BLOCK*\" +\n          \" internalReleaseLease: All existing blocks are COMPLETE,\" +\n          \" lease removed, file \" + src + \" closed.\");\n      return true;  // closed!\n    }\n\n    // Only the last and the penultimate blocks may be in non COMPLETE state.\n    // If the penultimate block is not COMPLETE, then it must be COMMITTED.\n    if(nrCompleteBlocks < nrBlocks - 2 ||\n       nrCompleteBlocks == nrBlocks - 2 &&\n         curBlock != null &&\n         curBlock.getBlockUCState() != BlockUCState.COMMITTED) {\n      final String message = \"DIR* NameSystem.internalReleaseLease: \"\n        + \"attempt to release a create lock on \"\n        + src + \" but file is already closed.\";\n      NameNode.stateChangeLog.warn(message);\n      throw new IOException(message);\n    }\n\n    // The last block is not COMPLETE, and\n    // that the penultimate block if exists is either COMPLETE or COMMITTED\n    final BlockInfo lastBlock = pendingFile.getLastBlock();\n    BlockUCState lastBlockState = lastBlock.getBlockUCState();\n    BlockInfo penultimateBlock = pendingFile.getPenultimateBlock();\n\n    // If penultimate block doesn't exist then its minReplication is met\n    boolean penultimateBlockMinStorage = penultimateBlock == null ||\n        blockManager.hasMinStorage(penultimateBlock);\n\n    switch(lastBlockState) {\n    case COMPLETE:\n      assert false : \"Already checked that the last block is incomplete\";\n      break;\n    case COMMITTED:\n      // Close file if committed blocks are minimally replicated","sourceCodeStart":3799,"sourceCodeEnd":3835,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L3799-L3835","documentation":"internalReleaseLease throws IOException when it is asked to release a create lock but the file's block states violate the open-file invariant: more than the last two blocks are non-COMPLETE, or the penultimate block is neither COMPLETE nor COMMITTED. Because the file looks effectively closed but inconsistent, lease recovery aborts with a DIR* state-change warning instead of corrupting the namespace.","triggerScenarios":"Lease expiry or recoverLease on a file where earlier (non-terminal) blocks were never committed: repeated failed pipeline recoveries, a crashed writer with multiple abandoned blocks, or block-state corruption from a buggy truncate/recovery path.","commonSituations":"Long-running writers killed mid-file whose pipelines repeatedly broke; files hit by known recovery bugs in older Hadoop 2.x releases; namespace restored from a bad checkpoint.","solutions":["Run 'hdfs fsck /path -files -blocks -locations' to see the block states of the affected file","If the data is dispensable, delete the file (delete also removes its lease) and rewrite it","If data must be saved, copy out readable bytes (hdfs dfs -get / distcp), then delete and re-ingest","Try 'hdfs debug recoverLease -path <path> -retries N' to force recovery attempts","Upgrade Hadoop if the block-state corruption matches a fixed JIRA (check truncate/recovery fixes for your version)"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  dfs.recoverLease(path);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"attempt to release a create lock\")) {\n    // invariant-broken file: salvage data then delete and rewrite\n    salvageAndRewrite(path);\n  } else { throw e; }\n}","preventionTips":["Keep Hadoop patched: several lease-recovery invariant bugs were fixed after 2.x","Avoid repeated force-kills of writers mid-pipeline; they leave abandoned block states","Periodically fsck files that survived crash cycles"],"tags":["hdfs","namenode","lease-recovery","block-state","invariant"],"backgroundTag":"lease-recovery-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}