apache/hadoop · warning · IOException

Got incremental block report from unregistered or dead node

Error message

Got incremental block report from unregistered or dead node

What it means

Thrown by BlockManager.processIncrementalBlockReport when a datanode's incremental report (received/deleted blocks) arrives from a nodeID that is unknown or not registered. Incremental reports mutate live block maps, so the NN rejects them from unregistered senders; as a self-healing measure the catch/finally sets node.setForceRegistration(true) so the next heartbeat re-registers the node.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:4654

    }
    return true;
  }

  /**
   * The given node is reporting incremental information about some blocks.
   * This includes blocks that are starting to be received, completed being
   * received, or deleted.
   * 
   * This method must be called with FSNamesystem lock held.
   */
  public void processIncrementalBlockReport(final DatanodeID nodeID,
      final StorageReceivedDeletedBlocks srdb) throws IOException {
    assert namesystem.hasWriteLock(RwLockMode.GLOBAL);
    final DatanodeDescriptor node = datanodeManager.getDatanode(nodeID);
    if (node == null || !node.isRegistered()) {
      blockLog.warn("BLOCK* processIncrementalBlockReport"
              + " is received from dead or unregistered node {}", nodeID);
      throw new IOException(
          "Got incremental block report from unregistered or dead node");
    }

    boolean successful = false;
    try {
      processIncrementalBlockReport(node, srdb);
      successful = true;
    } finally {
      if (!successful) {
        node.setForceRegistration(true);
      }
    }
  }

  private void processIncrementalBlockReport(final DatanodeDescriptor node,
      final StorageReceivedDeletedBlocks srdb) throws IOException {
    DatanodeStorageInfo storageInfo =
        node.getStorageInfo(srdb.getStorage().getStorageID());

View on GitHub (pinned to 2add963021)

Solutions

Mostly self-healing: forceRegistration is set, next heartbeat re-registers; monitor that the error stops after one or two heartbeat intervals
If it persists, check why registration is failing (include-list, storage ID conflicts, NN log 'registerDatanode' entries) and `hdfs dfsadmin -refreshNodes`
Restart the datanode to force a clean registration handshake
During planned NN restarts, expect a burst of these; they are benign if followed by successful registration

Example fix

# before: write pipeline fails during NN failover
# NN log: Got incremental block report from unregistered or dead node

# after: let heartbeat re-register (auto), verify with
hdfs dfsadmin -report | grep -A3 <host>
# if repeated: hdfs --daemon restart datanode

Defensive patterns

Strategy: retry

Try / catch

try {
  nn.blockReceivedAndDeleted(registration, poolId, srdb);
} catch (IOException e) {
  if (e.getMessage().contains("unregistered or dead node")) {
    // NN set forceRegistration; next heartbeat re-registers — retry IBR after heartbeat
    heartbeat(); retryBlockReceivedAndDeleted(srdb);
  } else { throw e; }
}

Prevention

Rely on built-in self-healing: forceRegistration triggers re-registration on heartbeat
During failovers, expect a short window of IBR rejections — build write retries accordingly
Keep DN heartbeat interval (dfs.heartbeat.interval) low on churn-heavy clusters
Verify with dfsadmin -report that the node eventually stays registered

When it happens

Trigger: DN sends IBR (e.g., receivedBlock acks during a pipeline write, or deleted-block acks) after its registration lapsed: NN restart/failover, node removed by admin, registration timeout; also a first IBR racing the registration RPC on DN startup.

Common situations: Writes failing right after NN failover with this error inside the pipeline heartbeat path; rolling upgrades; nodes under decommission still acking deletions.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1b2c0dbb03203d38. Report an issue: GitHub.