{"record":{"id":"5e5293318a0522b0","repo":"apache/hadoop","slug":"cannot-complete-block-block-does-not-satisfy-mini","errorCode":null,"errorMessage":"Cannot complete block: block does not satisfy minimal replication requirement.","messagePattern":"Cannot complete block: block does not satisfy minimal replication requirement\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java","lineNumber":1304,"sourceCode":"\n  /**\n   * Convert a specified block of the file to a complete block.\n   * @param curBlock - block to be completed\n   * @param iip - INodes in path to file containing curBlock; if null,\n   *              this will be resolved internally\n   * @param force - force completion of the block\n   * @throws IOException if the block does not have at least a minimal number\n   * of replicas reported from data-nodes.\n   */\n  private void completeBlock(BlockInfo curBlock, INodesInPath iip,\n      boolean force) throws IOException {\n    if (curBlock.isComplete()) {\n      return;\n    }\n\n    int numNodes = curBlock.numNodes();\n    if (!force && !hasMinStorage(curBlock, numNodes)) {\n      throw new IOException(\"Cannot complete block: \"\n          + \"block does not satisfy minimal replication requirement.\");\n    }\n    if (!force && curBlock.getBlockUCState() != BlockUCState.COMMITTED) {\n      throw new IOException(\n          \"Cannot complete block: block has not been COMMITTED by the client\");\n    }\n\n    convertToCompleteBlock(curBlock, iip);\n\n    // Since safe-mode only counts complete blocks, and we now have\n    // one more complete block, we need to adjust the total up, and\n    // also count it as safe, if we have at least the minimum replica\n    // count. (We may not have the minimum replica count yet if this is\n    // a \"forced\" completion when a file is getting closed by an\n    // OP_CLOSE edit on the standby).\n    bmSafeMode.adjustBlockTotals(0, 1);\n    final int minStorage = curBlock.isStriped() ?\n        ((BlockInfoStriped) curBlock).getRealDataBlockNum() : minReplication;","sourceCodeStart":1286,"sourceCodeEnd":1322,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L1286-L1322","documentation":"Thrown by BlockManager.completeBlock when the NameNode tries to transition a block from COMMITTED to COMPLETE but the number of reported replicas does not satisfy the minimal storage requirement (minReplication, or the EC policy's minimum for striped blocks; hasMinStorage returns false). The NameNode refuses to mark a block complete until enough live datanodes have reported holding final replicas, because a 'complete' block is trusted for reads and for safe-mode accounting.","triggerScenarios":"Client calls completeFile (non-forced path) while fewer than minReplication datanodes have reported the finalized replica via block reports/heartbeats — e.g., datanodes slow to send incremental block reports, replicas on stale/decommissioning/corrupt nodes, or cluster shrunk below dfs.namenode.replication.min.","commonSituations":"Cluster with fewer live datanodes than replication factor and some nodes dead/decommissioning; block reports delayed right after a big write burst; replicas marked corrupt; safe-mode edge cases at startup; EC file with fewer available data units.","solutions":["Retry completeFile — datanode incremental block reports typically arrive within seconds and the next attempt succeeds","Check cluster health: `hdfs dfsadmin -report` for live/dead datanodes, ensure live nodes >= dfs.namenode.replication.min (default 1) and ideally >= replication factor","Investigate why replicas are missing: corrupt-replica NN logs, under-replicated blocks via `hdfs fsck / -list-corruptfileblocks`, dead/stale nodes via datanode UI","As a last resort on a temporarily shrunken cluster, lower dfs.namenode.replication.min so files can close (trade-off: weaker durability guarantee at close time)"],"exampleFix":"// before: single close attempt fails when IBRs have not landed\nout.close(); // completeBlock -> min replication not met\n\n// after: retry loop; NN completes once DNs report the replica\ntry (FSDataOutputStream out = fs.append(src)) { out.write(...); }\nfor (int i = 0; i < 30 && !fs.isFileClosed(src); i++) { Thread.sleep(1000); }","handlingStrategy":"retry","validationCode":"// Pre-check: cluster must have >= min live replicas before closing\nDistributedFileSystem dfs = (DistributedFileSystem) fs;\nint live = dfs.getDataNodeStats(HdfsConstants.DatanodeReportTypes.LIVE).length;\nshort min = (short) dfs.getConf().getInt(DFSConfigKeys.DFS_NAMENODE_MIN_REPLICATION_KEY, 1);\nif (live < min) { throw new IOException(\"Cluster too small to close files safely\"); }","typeGuard":null,"tryCatchPattern":"try {\n  out.close();\n} catch (IOException e) {\n  if (e.getMessage().contains(\"minimal replication\")) {\n    for (int i = 0; i < 60 && !fs.isFileClosed(src); i++) { TimeUnit.SECONDS.sleep(1); } // IBRs land, NN auto-completes\n    if (!fs.isFileClosed(src)) { nn.recoverLease(src, clientName); }\n  } else { throw e; }\n}","preventionTips":["Keep live datanode count at or above the file's replication factor","Monitor under-replicated/corrupt blocks with fsck and NN JMX before they block file close","Retry close idempotently — completeFile is safe to re-call","Do not shrink clusters below dfs.namenode.replication.min while writes are open"],"tags":["hdfs","namenode","block","replication","min-replication","file-close"],"backgroundTag":"under-replicated-block","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}