apache/hadoop · error · IOException

Cannot complete block: block has not been COMMITTED by the c

Error message

Cannot complete block: block has not been COMMITTED by the client

What it means

Thrown by BlockManager.completeBlock when the NameNode is asked (without force) to complete a block whose under-construction state is not COMMITTED — i.e., the client never sent the finalize/commit for that block (it is still UNDER_CONSTRUCTION or UNDER_RECOVERY). Block lifecycle is UNDER_CONSTRUCTION -> COMMITTED -> COMPLETE; only the writing client performs the commit by flushing/finalizing the pipeline, so a complete request for an uncommitted block means the client skipped or lost that step.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:1308

   * @param iip - INodes in path to file containing curBlock; if null,
   *              this will be resolved internally
   * @param force - force completion of the block
   * @throws IOException if the block does not have at least a minimal number
   * of replicas reported from data-nodes.
   */
  private void completeBlock(BlockInfo curBlock, INodesInPath iip,
      boolean force) throws IOException {
    if (curBlock.isComplete()) {
      return;
    }

    int numNodes = curBlock.numNodes();
    if (!force && !hasMinStorage(curBlock, numNodes)) {
      throw new IOException("Cannot complete block: "
          + "block does not satisfy minimal replication requirement.");
    }
    if (!force && curBlock.getBlockUCState() != BlockUCState.COMMITTED) {
      throw new IOException(
          "Cannot complete block: block has not been COMMITTED by the client");
    }

    convertToCompleteBlock(curBlock, iip);

    // Since safe-mode only counts complete blocks, and we now have
    // one more complete block, we need to adjust the total up, and
    // also count it as safe, if we have at least the minimum replica
    // count. (We may not have the minimum replica count yet if this is
    // a "forced" completion when a file is getting closed by an
    // OP_CLOSE edit on the standby).
    bmSafeMode.adjustBlockTotals(0, 1);
    final int minStorage = curBlock.isStriped() ?
        ((BlockInfoStriped) curBlock).getRealDataBlockNum() : minReplication;
    bmSafeMode.incrementSafeBlockCount(Math.min(numNodes, minStorage),
        curBlock);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Use recoverLease instead of a direct complete call for orphaned writes: `hdfs debug recoverLease -path <file> -retries 3` lets NN finalize via recovery with the correct state machine
2. Fix the writing client so it always closes its output stream (try-with-resources) so the normal commit-then-complete sequence runs
3. Check NN block log for the file's last block state (`hdfs fsck <file> -files -blocks`) to confirm it is stuck UNDER_CONSTRUCTION
4. After recovery closes the file, open a new append if more data must be written

Example fix

# before: file left open after writer crash, naive close attempt
hdfs dfs -appendToFile data file  # or client completeFile -> not COMMITTED

# after: recover the lease so NN drives the block to COMPLETE
hdfs debug recoverLease -path /path/file -retries 5
hdfs fsck /path/file -files -blocks   # verify block state COMPLETE

// Before closing manually, confirm the last block is finalized
HdfsLocatedFileStatus st = (HdfsLocatedFileStatus) fs.getFileStatus(src);
boolean lastBlockCommitted = st.isUnderConstruction() == false
    || st.getLastLocatedBlock() != null && st.getLastLocatedBlock().isCorrupt() == false;
// If writer crashed with block UNDER_CONSTRUCTION, go through recovery, not complete

try {
  nn.complete(src, clientName, lastBlock, iip);
} catch (IOException e) {
  if (e.getMessage().contains("not been COMMITTED")) {
    nn.recoverLease(src, clientName);       // NN finalizes via recovery state machine
    while (!fs.isFileClosed(src)) { TimeUnit.SECONDS.sleep(1); }
  } else { throw e; }
}

When it happens

Trigger: completeFile invoked for a file whose last block was allocated but never finalized by the client (client crashed after addBlock, before hflush/close); calling NN internals completeFile with inconsistent state; client retry that bypasses the commit handshake; replayed edit log on standby uses force=true so it avoids this — the non-forced RPC path does not.

Common situations: Application killed mid-write and something (script, second client) tries to close the file directly; buggy custom client calling completeFile without writing/finalizing; append opened but nothing written; lease recovery interleaving leaving block in UNDER_RECOVERY.

Related errors

• Commit or complete block {commitBlock}, whereas it is under
• Commit block with mismatching GS. NN has {block}, client sub
• Cannot complete block: block does not satisfy minimal replic
• Recovery block {b} where it is not under construction.
• DIR* NameSystem.internalReleaseLease: attempt to release a c

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/979965df36ffafdc. Report an issue: GitHub.