{"record":{"id":"df8f6a1b775ebd6b","repo":"apache/hadoop","slug":"recovery-block-b-where-it-is-not-under-construct","errorCode":null,"errorMessage":"Recovery block {b} where it is not under construction.","messagePattern":"Recovery block (.+?) where it is not under construction\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java","lineNumber":1762,"sourceCode":"    return !hostname.equals(ip) || NetUtils.isLocalAddress(address);\n  }\n  \n  private void setDatanodeDead(DatanodeDescriptor node) {\n    node.setLastUpdate(0);\n    node.setLastUpdateMonotonic(0);\n  }\n\n  private BlockRecoveryCommand getBlockRecoveryCommand(String blockPoolId,\n      DatanodeDescriptor nodeinfo) throws IOException {\n    BlockInfo[] blocks = nodeinfo.getLeaseRecoveryCommand(Integer.MAX_VALUE);\n    if (blocks == null) {\n      return null;\n    }\n    BlockRecoveryCommand brCommand = new BlockRecoveryCommand(blocks.length);\n    for (BlockInfo b : blocks) {\n      BlockUnderConstructionFeature uc = b.getUnderConstructionFeature();\n      if(uc == null) {\n        throw new IOException(\"Recovery block \" + b +\n            \" where it is not under construction.\");\n      }\n      final DatanodeStorageInfo[] storages = uc.getExpectedStorageLocations();\n      // Skip stale and dead nodes during recovery.\n      List<DatanodeStorageInfo> recoveryLocations =\n          new ArrayList<>(storages.length);\n      List<Integer> storageIdx = new ArrayList<>(storages.length);\n      for (int i = 0; i < storages.length; ++i) {\n        if (!storages[i].getDatanodeDescriptor().isStale(staleInterval) &&\n            storages[i].getDatanodeDescriptor().isAlive()) {\n          recoveryLocations.add(storages[i]);\n          storageIdx.add(i);\n        }\n      }\n      // If we are performing a truncate recovery than set recovery fields\n      // to old block.\n      boolean truncateRecovery = uc.getTruncateBlock() != null;\n      boolean copyOnTruncateRecovery = truncateRecovery &&","sourceCodeStart":1744,"sourceCodeEnd":1780,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java#L1744-L1780","documentation":"Thrown by DatanodeManager.getBlockRecoveryCommand when a block queued for lease recovery on a datanode (nodeinfo.getLeaseRecoveryCommand) has no BlockUnderConstructionFeature — i.e., the NameNode considers the block finalized/complete, not under construction. Recovery commands are only valid for blocks with expected replica locations recorded in the UC feature; a complete block reaching this loop means internal state desync between the lease-recovery queue and block state.","triggerScenarios":"Heartbeat path: DN asks for nextHeartbeat commands, NN iterates leaseRecoveryCommand blocks for the node and one block has uc == null — can follow a race where block finalization (commit/complete) happened while the recovery command was still queued, or NN state inconsistencies after failover/replay.","commonSituations":"Rare internal race during concurrent lease recovery and file close; NN crash-recovery/HA failover replaying inconsistent edit logs; almost always a NameNode-side bug-worthy condition rather than client misconfiguration — seen historically around recovery-during-complete races (fixed by HDFS patches like HDFS-5185-era work).","solutions":["Check the NN version for known recovery races and upgrade to a release containing lease-recovery/complete race fixes (e.g., 2.4+ lineage patches)","Capture NN log with the block id and file a bug / search Hadoop JIRA for the block signature if reproducible","Restarting the NameNode clears the in-memory recovery queue desync as a mitigation","Retry the client operation after NN restart; if the file is stuck, recoverLease to force a clean close"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// NN heartbeat-handler context (or test harness): isolate internal desync\ntry {\n  cmd = getBlockRecoveryCommand(poolId, nodeinfo);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"not under construction\")) {\n    LOG.warn(\"Lease recovery queue desync for {} — recovering via block state\", nodeinfo);\n    // re-run lease recovery: recoverLease triggers initiateFileRecovery fresh state\n  } else { throw e; }\n}","preventionTips":["Run a current Hadoop release with lease-recovery race fixes; avoid EOL lines","Restart NN to clear in-memory desync when this appears after failover","Capture block id + JIRA-search the signature; report if novel","Use recoverLease rather than manual close when files are wedged"],"tags":["hdfs","lease-recovery","block-state","namenode","race-condition","internal-error"],"backgroundTag":"lease-recovery-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}