apache/hadoop · error · AlreadyBeingCreatedException

DIR* NameSystem.internalReleaseLease: Failed to release leas

Error message

DIR* NameSystem.internalReleaseLease: Failed to release lease for file {}. Committed blocks are waiting to be minimally replicated.

What it means

internalReleaseLease cannot close the file because the committed last block has not reached minimal replication and the penultimate block is also below min storage. It throws AlreadyBeingCreatedException so lease recovery is retried later; the log explicitly warns that recovery can loop forever if no valid replicas exist on the datanodes.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:3854

      if(penultimateBlockMinStorage &&
          blockManager.hasMinStorage(lastBlock)) {
        finalizeINodeFileUnderConstruction(src, pendingFile,
            iip.getLatestSnapshotId(), false);
        NameNode.stateChangeLog.warn("BLOCK*" +
            " internalReleaseLease: Committed blocks are minimally" +
            " replicated, lease removed, file" + src + " closed.");
        return true;  // closed!
      }
      // Cannot close file right now, since some blocks 
      // are not yet minimally replicated.
      // This may potentially cause infinite loop in lease recovery
      // if there are no valid replicas on data-nodes.
      String message = "DIR* NameSystem.internalReleaseLease: " +
          "Failed to release lease for file " + src +
          ". Committed blocks are waiting to be minimally replicated.";
      NameNode.stateChangeLog.warn(message);
      if (!penultimateBlockMinStorage) {
        throw new AlreadyBeingCreatedException(message);
      }
      // Intentionally fall through to UNDER_RECOVERY so BLOCK_RECOVERY is
      // attempted
    case UNDER_CONSTRUCTION:
    case UNDER_RECOVERY:
      BlockUnderConstructionFeature uc =
          lastBlock.getUnderConstructionFeature();
      // determine if last block was intended to be truncated
      BlockInfo recoveryBlock = uc.getTruncateBlock();
      boolean truncateRecovery = recoveryBlock != null;
      boolean copyOnTruncate = truncateRecovery &&
          recoveryBlock.getBlockId() != lastBlock.getBlockId();
      assert !copyOnTruncate ||
          recoveryBlock.getBlockId() < lastBlock.getBlockId() &&
          recoveryBlock.getGenerationStamp() < lastBlock.getGenerationStamp() &&
          recoveryBlock.getNumBytes() > lastBlock.getNumBytes() :
            "wrong recoveryBlock";

View on GitHub (pinned to 2add963021)

Solutions

1. Restore the datanodes that hold replicas (restart them, fix mounts) so replication can be met, then retry recovery
2. Use 'hdfs debug recoverLease -path <path> -retries N' to drive repeated attempts
3. Check 'hdfs dfsadmin -report' and under-replicated block count; fix under-replication first
4. If replicas are permanently lost, salvage readable bytes (dfs -get), delete the file, and re-ingest

try {
  dfs.recoverLease(path);
} catch (AlreadyBeingCreatedException e) {
  // blocks under-replicated: wait for datanodes/replication, then retry
  waitForMinReplication(path, timeout);
  dfs.recoverLease(path);
}

When it happens

Trigger: Lease recovery while the last (COMMITTED) block and/or penultimate block are under-replicated: pipeline datanodes died before commit, replicas lost to decommissioning or disk failures.

Common situations: Writer crashed and its pipeline DNs went down in the same incident; small clusters where one dead DN drops replication below minReplication; flaky disks losing replica files.

Related errors

• Replica was found but missing fields.
• Trying to commit inconsistent block: id = {blockId}, expecte
• Commit block with mismatching GS. NN has {block}, client sub
• Commit or complete block {commitBlock}, whereas it is under
• Cannot complete block: block has not been COMMITTED by the c

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d8caf1ffb0cad07. Report an issue: GitHub.