{"record":{"id":"169f595e4b34911b","repo":"apache/hadoop","slug":"commit-or-complete-block-commitblock-whereas-it","errorCode":null,"errorMessage":"Commit or complete block {commitBlock}, whereas it is under recovery.","messagePattern":"Commit or complete block (.+?), whereas it is under recovery\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java","lineNumber":1222,"sourceCode":"   * \n   * @param bc block collection\n   * @param commitBlock - contains client reported block length and generation\n   * @param iip - INodes in path to bc\n   * @return true if the last block is changed to committed state.\n   * @throws IOException if the block does not have at least a minimal number\n   * of replicas reported from data-nodes.\n   */\n  public boolean commitOrCompleteLastBlock(BlockCollection bc,\n      Block commitBlock, INodesInPath iip) throws IOException {\n    if(commitBlock == null)\n      return false; // not committing, this is a block allocation retry\n    BlockInfo lastBlock = bc.getLastBlock();\n    if(lastBlock == null)\n      return false; // no blocks in file yet\n    if(lastBlock.isComplete())\n      return false; // already completed (e.g. by syncBlock)\n    if(lastBlock.isUnderRecovery()) {\n      throw new IOException(\"Commit or complete block \" + commitBlock +\n          \", whereas it is under recovery.\");\n    }\n    \n    final boolean committed = commitBlock(lastBlock, commitBlock);\n    if (committed && lastBlock.isStriped()) {\n      // update scheduled size for DatanodeStorages that do not store any\n      // internal blocks\n      lastBlock.getUnderConstructionFeature()\n          .updateStorageScheduledSize((BlockInfoStriped) lastBlock);\n    }\n\n    // Count replicas on decommissioning nodes, as these will not be\n    // decommissioned unless recovery/completing last block has finished\n    NumberReplicas numReplicas = countNodes(lastBlock);\n    int numUsableReplicas = numReplicas.liveReplicas() +\n        numReplicas.decommissioning() +\n        numReplicas.liveEnteringMaintenanceReplicas();\n","sourceCodeStart":1204,"sourceCodeEnd":1240,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L1204-L1240","documentation":"Thrown by BlockManager.commitOrCompleteLastBlock when a client tries to commit or complete the last block of a file whose block is currently under lease recovery (BlockInfo.isUnderRecovery() is true). While recovery is in progress the NameNode freezes normal commit/complete transitions because the recovery process itself will finalize the block with a new generation stamp. Committing concurrently would race with the recovery and corrupt block bookkeeping.","triggerScenarios":"Client calls completeFile/addBlock-commit path while another client or the NameNode's LeaseMonitor triggered recoverLease on the same file; block recovery was initiated (initiateFileRecovery) and has not received all replica recovery acks yet; the original writer resumes and calls complete() during that window.","commonSituations":"Original writer slow/blocked (GC, network hiccup) past the soft lease limit while a reader or job driver called recoverLease; two frameworks (e.g., Hive + distcp) touching the same file; hard lease expiry during long appends; failover of an app between HA NameNodes with in-flight writes.","solutions":["Let the recovery finish: poll recoverLease/completeFile until it returns true or the file shows as closed — the NN finalizes the block via recovery","Have the competing reader/job stop calling recoverLease on a file the writer still owns; fix the writer so it heartbeats/renews its lease (avoid long GC pauses)","If the writer is genuinely dead, abandon its handle and let recovery close the file, then rewrite/append as a new write","Verify only one client holds the write lease (NN log shows 'Recovering lease' entries)"],"exampleFix":"// before: single completeFile call races active lease recovery\nboolean done = namenode.complete(src, clientName, lastBlock, iip); // throws: under recovery\n\n// after: drain recovery first, then complete\nif (!namenode.complete(src, clientName, lastBlock, iip)) {\n  namenode.recoverLease(src, clientName);\n  while (!namenode.complete(src, clientName, lastBlock, iip)) {\n    Thread.sleep(1000); // recovery finalizes the block; then complete returns true\n  }\n}","handlingStrategy":"retry","validationCode":"// Only attempt complete when the file is not actively under recovery\nboolean underRecovery = false;\nfor (LocatedBlock lb : ((DistributedFileSystem) fs).listLocatedStatus(src) instanceof HdfsLocatedFileStatus s\n    ? Iterables.toArray(s.getLocatedBlocks(), LocatedBlock.class) : new LocatedBlock[0]) {\n  underRecovery |= lb.isUnderRecovery() && lb == last;\n}\nif (!underRecovery) { nn.complete(src, clientName, lastBlock, iip); }","typeGuard":null,"tryCatchPattern":"try {\n  nn.complete(src, clientName, lastBlock, iip);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"under recovery\")) {\n    // recovery will finalize the block; poll for file closure\n    while (!fs.isFileClosed(src)) { TimeUnit.SECONDS.sleep(1); }\n  } else { throw e; }\n}","preventionTips":["Close files promptly; do not hold write streams idle past the soft lease limit (dfs.namenode.lease-hard-checked-interval)","Prevent competing recoverLease calls while a writer is alive","Heartbeat via hflush on long appends to keep the lease renewed","After failover, check lease state before resuming writes"],"tags":["hdfs","namenode","block","lease-recovery","concurrency","file-close"],"backgroundTag":"lease-recovery-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}