apache/hadoop · error · IOException

Cannot complete block: block does not satisfy minimal replic

Error message

Cannot complete block: block does not satisfy minimal replication requirement.

What it means

Thrown by BlockManager.completeBlock when the NameNode tries to transition a block from COMMITTED to COMPLETE but the number of reported replicas does not satisfy the minimal storage requirement (minReplication, or the EC policy's minimum for striped blocks; hasMinStorage returns false). The NameNode refuses to mark a block complete until enough live datanodes have reported holding final replicas, because a 'complete' block is trusted for reads and for safe-mode accounting.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:1304

  /**
   * Convert a specified block of the file to a complete block.
   * @param curBlock - block to be completed
   * @param iip - INodes in path to file containing curBlock; if null,
   *              this will be resolved internally
   * @param force - force completion of the block
   * @throws IOException if the block does not have at least a minimal number
   * of replicas reported from data-nodes.
   */
  private void completeBlock(BlockInfo curBlock, INodesInPath iip,
      boolean force) throws IOException {
    if (curBlock.isComplete()) {
      return;
    }

    int numNodes = curBlock.numNodes();
    if (!force && !hasMinStorage(curBlock, numNodes)) {
      throw new IOException("Cannot complete block: "
          + "block does not satisfy minimal replication requirement.");
    }
    if (!force && curBlock.getBlockUCState() != BlockUCState.COMMITTED) {
      throw new IOException(
          "Cannot complete block: block has not been COMMITTED by the client");
    }

    convertToCompleteBlock(curBlock, iip);

    // Since safe-mode only counts complete blocks, and we now have
    // one more complete block, we need to adjust the total up, and
    // also count it as safe, if we have at least the minimum replica
    // count. (We may not have the minimum replica count yet if this is
    // a "forced" completion when a file is getting closed by an
    // OP_CLOSE edit on the standby).
    bmSafeMode.adjustBlockTotals(0, 1);
    final int minStorage = curBlock.isStriped() ?
        ((BlockInfoStriped) curBlock).getRealDataBlockNum() : minReplication;

View on GitHub (pinned to 2add963021)

Solutions

1. Retry completeFile — datanode incremental block reports typically arrive within seconds and the next attempt succeeds
2. Check cluster health: `hdfs dfsadmin -report` for live/dead datanodes, ensure live nodes >= dfs.namenode.replication.min (default 1) and ideally >= replication factor
3. Investigate why replicas are missing: corrupt-replica NN logs, under-replicated blocks via `hdfs fsck / -list-corruptfileblocks`, dead/stale nodes via datanode UI
4. As a last resort on a temporarily shrunken cluster, lower dfs.namenode.replication.min so files can close (trade-off: weaker durability guarantee at close time)

Example fix

// before: single close attempt fails when IBRs have not landed
out.close(); // completeBlock -> min replication not met

// after: retry loop; NN completes once DNs report the replica
try (FSDataOutputStream out = fs.append(src)) { out.write(...); }
for (int i = 0; i < 30 && !fs.isFileClosed(src); i++) { Thread.sleep(1000); }

// Pre-check: cluster must have >= min live replicas before closing
DistributedFileSystem dfs = (DistributedFileSystem) fs;
int live = dfs.getDataNodeStats(HdfsConstants.DatanodeReportTypes.LIVE).length;
short min = (short) dfs.getConf().getInt(DFSConfigKeys.DFS_NAMENODE_MIN_REPLICATION_KEY, 1);
if (live < min) { throw new IOException("Cluster too small to close files safely"); }

try {
  out.close();
} catch (IOException e) {
  if (e.getMessage().contains("minimal replication")) {
    for (int i = 0; i < 60 && !fs.isFileClosed(src); i++) { TimeUnit.SECONDS.sleep(1); } // IBRs land, NN auto-completes
    if (!fs.isFileClosed(src)) { nn.recoverLease(src, clientName); }
  } else { throw e; }
}

When it happens

Trigger: Client calls completeFile (non-forced path) while fewer than minReplication datanodes have reported the finalized replica via block reports/heartbeats — e.g., datanodes slow to send incremental block reports, replicas on stale/decommissioning/corrupt nodes, or cluster shrunk below dfs.namenode.replication.min.

Common situations: Cluster with fewer live datanodes than replication factor and some nodes dead/decommissioning; block reports delayed right after a big write burst; replicas marked corrupt; safe-mode edge cases at startup; EC file with fewer available data units.

Related errors

• Commit or complete block {commitBlock}, whereas it is under
• Cannot complete block: block has not been COMMITTED by the c
• Unexpected configuration parameters: dfs.namenode.replicatio
• Unexpected configuration parameters: dfs.replication.max = {
• Unexpected configuration parameters: dfs.namenode.replicatio

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5e5293318a0522b0. Report an issue: GitHub.