apache/hadoop · error · IOException

Commit or complete block {commitBlock}, whereas it is under

Error message

Commit or complete block {commitBlock}, whereas it is under recovery.

What it means

Thrown by BlockManager.commitOrCompleteLastBlock when a client tries to commit or complete the last block of a file whose block is currently under lease recovery (BlockInfo.isUnderRecovery() is true). While recovery is in progress the NameNode freezes normal commit/complete transitions because the recovery process itself will finalize the block with a new generation stamp. Committing concurrently would race with the recovery and corrupt block bookkeeping.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:1222

   * 
   * @param bc block collection
   * @param commitBlock - contains client reported block length and generation
   * @param iip - INodes in path to bc
   * @return true if the last block is changed to committed state.
   * @throws IOException if the block does not have at least a minimal number
   * of replicas reported from data-nodes.
   */
  public boolean commitOrCompleteLastBlock(BlockCollection bc,
      Block commitBlock, INodesInPath iip) throws IOException {
    if(commitBlock == null)
      return false; // not committing, this is a block allocation retry
    BlockInfo lastBlock = bc.getLastBlock();
    if(lastBlock == null)
      return false; // no blocks in file yet
    if(lastBlock.isComplete())
      return false; // already completed (e.g. by syncBlock)
    if(lastBlock.isUnderRecovery()) {
      throw new IOException("Commit or complete block " + commitBlock +
          ", whereas it is under recovery.");
    }
    
    final boolean committed = commitBlock(lastBlock, commitBlock);
    if (committed && lastBlock.isStriped()) {
      // update scheduled size for DatanodeStorages that do not store any
      // internal blocks
      lastBlock.getUnderConstructionFeature()
          .updateStorageScheduledSize((BlockInfoStriped) lastBlock);
    }

    // Count replicas on decommissioning nodes, as these will not be
    // decommissioned unless recovery/completing last block has finished
    NumberReplicas numReplicas = countNodes(lastBlock);
    int numUsableReplicas = numReplicas.liveReplicas() +
        numReplicas.decommissioning() +
        numReplicas.liveEnteringMaintenanceReplicas();

View on GitHub (pinned to 2add963021)

Solutions

Let the recovery finish: poll recoverLease/completeFile until it returns true or the file shows as closed — the NN finalizes the block via recovery
Have the competing reader/job stop calling recoverLease on a file the writer still owns; fix the writer so it heartbeats/renews its lease (avoid long GC pauses)
If the writer is genuinely dead, abandon its handle and let recovery close the file, then rewrite/append as a new write
Verify only one client holds the write lease (NN log shows 'Recovering lease' entries)

Example fix

// before: single completeFile call races active lease recovery
boolean done = namenode.complete(src, clientName, lastBlock, iip); // throws: under recovery

// after: drain recovery first, then complete
if (!namenode.complete(src, clientName, lastBlock, iip)) {
  namenode.recoverLease(src, clientName);
  while (!namenode.complete(src, clientName, lastBlock, iip)) {
    Thread.sleep(1000); // recovery finalizes the block; then complete returns true
  }
}

Defensive patterns

Strategy: retry

Validate before calling

// Only attempt complete when the file is not actively under recovery
boolean underRecovery = false;
for (LocatedBlock lb : ((DistributedFileSystem) fs).listLocatedStatus(src) instanceof HdfsLocatedFileStatus s
    ? Iterables.toArray(s.getLocatedBlocks(), LocatedBlock.class) : new LocatedBlock[0]) {
  underRecovery |= lb.isUnderRecovery() && lb == last;
}
if (!underRecovery) { nn.complete(src, clientName, lastBlock, iip); }

Try / catch

try {
  nn.complete(src, clientName, lastBlock, iip);
} catch (IOException e) {
  if (e.getMessage().contains("under recovery")) {
    // recovery will finalize the block; poll for file closure
    while (!fs.isFileClosed(src)) { TimeUnit.SECONDS.sleep(1); }
  } else { throw e; }
}

Prevention

Close files promptly; do not hold write streams idle past the soft lease limit (dfs.namenode.lease-hard-checked-interval)
Prevent competing recoverLease calls while a writer is alive
Heartbeat via hflush on long appends to keep the lease renewed
After failover, check lease state before resuming writes

When it happens

Trigger: Client calls completeFile/addBlock-commit path while another client or the NameNode's LeaseMonitor triggered recoverLease on the same file; block recovery was initiated (initiateFileRecovery) and has not received all replica recovery acks yet; the original writer resumes and calls complete() during that window.

Common situations: Original writer slow/blocked (GC, network hiccup) past the soft lease limit while a reader or job driver called recoverLease; two frameworks (e.g., Hive + distcp) touching the same file; hard lease expiry during long appends; failover of an app between HA NameNodes with in-flight writes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/169f595e4b34911b. Report an issue: GitHub.