apache/hadoop · error · IOException

Recovery block {b} where it is not under construction.

Error message

Recovery block {b} where it is not under construction.

What it means

Thrown by DatanodeManager.getBlockRecoveryCommand when a block queued for lease recovery on a datanode (nodeinfo.getLeaseRecoveryCommand) has no BlockUnderConstructionFeature — i.e., the NameNode considers the block finalized/complete, not under construction. Recovery commands are only valid for blocks with expected replica locations recorded in the UC feature; a complete block reaching this loop means internal state desync between the lease-recovery queue and block state.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:1762

    return !hostname.equals(ip) || NetUtils.isLocalAddress(address);
  }
  
  private void setDatanodeDead(DatanodeDescriptor node) {
    node.setLastUpdate(0);
    node.setLastUpdateMonotonic(0);
  }

  private BlockRecoveryCommand getBlockRecoveryCommand(String blockPoolId,
      DatanodeDescriptor nodeinfo) throws IOException {
    BlockInfo[] blocks = nodeinfo.getLeaseRecoveryCommand(Integer.MAX_VALUE);
    if (blocks == null) {
      return null;
    }
    BlockRecoveryCommand brCommand = new BlockRecoveryCommand(blocks.length);
    for (BlockInfo b : blocks) {
      BlockUnderConstructionFeature uc = b.getUnderConstructionFeature();
      if(uc == null) {
        throw new IOException("Recovery block " + b +
            " where it is not under construction.");
      }
      final DatanodeStorageInfo[] storages = uc.getExpectedStorageLocations();
      // Skip stale and dead nodes during recovery.
      List<DatanodeStorageInfo> recoveryLocations =
          new ArrayList<>(storages.length);
      List<Integer> storageIdx = new ArrayList<>(storages.length);
      for (int i = 0; i < storages.length; ++i) {
        if (!storages[i].getDatanodeDescriptor().isStale(staleInterval) &&
            storages[i].getDatanodeDescriptor().isAlive()) {
          recoveryLocations.add(storages[i]);
          storageIdx.add(i);
        }
      }
      // If we are performing a truncate recovery than set recovery fields
      // to old block.
      boolean truncateRecovery = uc.getTruncateBlock() != null;
      boolean copyOnTruncateRecovery = truncateRecovery &&

View on GitHub (pinned to 2add963021)

Solutions

Check the NN version for known recovery races and upgrade to a release containing lease-recovery/complete race fixes (e.g., 2.4+ lineage patches)
Capture NN log with the block id and file a bug / search Hadoop JIRA for the block signature if reproducible
Restarting the NameNode clears the in-memory recovery queue desync as a mitigation
Retry the client operation after NN restart; if the file is stuck, recoverLease to force a clean close

Defensive patterns

Strategy: try-catch

Try / catch

// NN heartbeat-handler context (or test harness): isolate internal desync
try {
  cmd = getBlockRecoveryCommand(poolId, nodeinfo);
} catch (IOException e) {
  if (e.getMessage().contains("not under construction")) {
    LOG.warn("Lease recovery queue desync for {} — recovering via block state", nodeinfo);
    // re-run lease recovery: recoverLease triggers initiateFileRecovery fresh state
  } else { throw e; }
}

Prevention

Run a current Hadoop release with lease-recovery race fixes; avoid EOL lines
Restart NN to clear in-memory desync when this appears after failover
Capture block id + JIRA-search the signature; report if novel
Use recoverLease rather than manual close when files are wedged

When it happens

Trigger: Heartbeat path: DN asks for nextHeartbeat commands, NN iterates leaseRecoveryCommand blocks for the node and one block has uc == null — can follow a race where block finalization (commit/complete) happened while the recovery command was still queued, or NN state inconsistencies after failover/replay.

Common situations: Rare internal race during concurrent lease recovery and file close; NN crash-recovery/HA failover replaying inconsistent edit logs; almost always a NameNode-side bug-worthy condition rather than client misconfiguration — seen historically around recovery-during-complete races (fixed by HDFS patches like HDFS-5185-era work).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/df8f6a1b775ebd6b. Report an issue: GitHub.