{"record":{"id":"979965df36ffafdc","repo":"apache/hadoop","slug":"cannot-complete-block-block-has-not-been-committe","errorCode":null,"errorMessage":"Cannot complete block: block has not been COMMITTED by the client","messagePattern":"Cannot complete block: block has not been COMMITTED by the client","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java","lineNumber":1308,"sourceCode":"   * @param iip - INodes in path to file containing curBlock; if null,\n   *              this will be resolved internally\n   * @param force - force completion of the block\n   * @throws IOException if the block does not have at least a minimal number\n   * of replicas reported from data-nodes.\n   */\n  private void completeBlock(BlockInfo curBlock, INodesInPath iip,\n      boolean force) throws IOException {\n    if (curBlock.isComplete()) {\n      return;\n    }\n\n    int numNodes = curBlock.numNodes();\n    if (!force && !hasMinStorage(curBlock, numNodes)) {\n      throw new IOException(\"Cannot complete block: \"\n          + \"block does not satisfy minimal replication requirement.\");\n    }\n    if (!force && curBlock.getBlockUCState() != BlockUCState.COMMITTED) {\n      throw new IOException(\n          \"Cannot complete block: block has not been COMMITTED by the client\");\n    }\n\n    convertToCompleteBlock(curBlock, iip);\n\n    // Since safe-mode only counts complete blocks, and we now have\n    // one more complete block, we need to adjust the total up, and\n    // also count it as safe, if we have at least the minimum replica\n    // count. (We may not have the minimum replica count yet if this is\n    // a \"forced\" completion when a file is getting closed by an\n    // OP_CLOSE edit on the standby).\n    bmSafeMode.adjustBlockTotals(0, 1);\n    final int minStorage = curBlock.isStriped() ?\n        ((BlockInfoStriped) curBlock).getRealDataBlockNum() : minReplication;\n    bmSafeMode.incrementSafeBlockCount(Math.min(numNodes, minStorage),\n        curBlock);\n  }\n","sourceCodeStart":1290,"sourceCodeEnd":1326,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L1290-L1326","documentation":"Thrown by BlockManager.completeBlock when the NameNode is asked (without force) to complete a block whose under-construction state is not COMMITTED — i.e., the client never sent the finalize/commit for that block (it is still UNDER_CONSTRUCTION or UNDER_RECOVERY). Block lifecycle is UNDER_CONSTRUCTION -> COMMITTED -> COMPLETE; only the writing client performs the commit by flushing/finalizing the pipeline, so a complete request for an uncommitted block means the client skipped or lost that step.","triggerScenarios":"completeFile invoked for a file whose last block was allocated but never finalized by the client (client crashed after addBlock, before hflush/close); calling NN internals completeFile with inconsistent state; client retry that bypasses the commit handshake; replayed edit log on standby uses force=true so it avoids this — the non-forced RPC path does not.","commonSituations":"Application killed mid-write and something (script, second client) tries to close the file directly; buggy custom client calling completeFile without writing/finalizing; append opened but nothing written; lease recovery interleaving leaving block in UNDER_RECOVERY.","solutions":["Use recoverLease instead of a direct complete call for orphaned writes: `hdfs debug recoverLease -path <file> -retries 3` lets NN finalize via recovery with the correct state machine","Fix the writing client so it always closes its output stream (try-with-resources) so the normal commit-then-complete sequence runs","Check NN block log for the file's last block state (`hdfs fsck <file> -files -blocks`) to confirm it is stuck UNDER_CONSTRUCTION","After recovery closes the file, open a new append if more data must be written"],"exampleFix":"# before: file left open after writer crash, naive close attempt\nhdfs dfs -appendToFile data file  # or client completeFile -> not COMMITTED\n\n# after: recover the lease so NN drives the block to COMPLETE\nhdfs debug recoverLease -path /path/file -retries 5\nhdfs fsck /path/file -files -blocks   # verify block state COMPLETE","handlingStrategy":"fallback","validationCode":"// Before closing manually, confirm the last block is finalized\nHdfsLocatedFileStatus st = (HdfsLocatedFileStatus) fs.getFileStatus(src);\nboolean lastBlockCommitted = st.isUnderConstruction() == false\n    || st.getLastLocatedBlock() != null && st.getLastLocatedBlock().isCorrupt() == false;\n// If writer crashed with block UNDER_CONSTRUCTION, go through recovery, not complete","typeGuard":null,"tryCatchPattern":"try {\n  nn.complete(src, clientName, lastBlock, iip);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"not been COMMITTED\")) {\n    nn.recoverLease(src, clientName);       // NN finalizes via recovery state machine\n    while (!fs.isFileClosed(src)) { TimeUnit.SECONDS.sleep(1); }\n  } else { throw e; }\n}","preventionTips":["Always close output streams with try-with-resources so commit-then-complete runs","Never call completeFile for a file you did not write in this session; use recoverLease","After a writer crash, prefer `hdfs debug recoverLease` over manual close APIs","Test crash-recovery paths in applications that write to HDFS"],"tags":["hdfs","namenode","block","block-state","file-close","lease-recovery"],"backgroundTag":"block-state-machine-violation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}