apache/hadoop · warning · IOException

ProcessReport from dead or unregistered node: {nodeID}

Error message

ProcessReport from dead or unregistered node: {nodeID}

What it means

Thrown by BlockManager.processReport (full block report handling) when the reporting nodeID resolves to null or a DatanodeDescriptor with isRegistered()==false. The NameNode only accepts full block reports from registered, live datanodes; anything else is rejected outright because block --> storage maps can only be updated for known storage identities.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:2931

   */
  public boolean processReport(final DatanodeID nodeID,
      final DatanodeStorage storage,
      final BlockListAsLongs newReport,
      BlockReportContext context) throws IOException {
    namesystem.writeLock(RwLockMode.GLOBAL);
    final long startTime = Time.monotonicNow(); //after acquiring write lock
    final long endTime;
    DatanodeDescriptor node;
    Collection<Block> invalidatedBlocks = Collections.emptyList();
    String strBlockReportId =
        context != null ? Long.toHexString(context.getReportId()) : "";
    String fullBrLeaseId =
        context != null ? Long.toHexString(context.getLeaseId()) : "";

    try {
      node = datanodeManager.getDatanode(nodeID);
      if (node == null || !node.isRegistered()) {
        throw new IOException(
            "ProcessReport from dead or unregistered node: " + nodeID);
      }

      // To minimize startup time, we discard any second (or later) block reports
      // that we receive while still in startup phase.
      // Register DN with provided storage, not with storage owned by DN
      // DN should still have a ref to the DNStorageInfo.
      DatanodeStorageInfo storageInfo =
          providedStorageMap.getStorage(node, storage);

      if (storageInfo == null) {
        // We handle this for backwards compatibility.
        storageInfo = node.updateStorage(storage);
      }
      if (namesystem.isInStartupSafeMode()
          && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
          && storageInfo.getBlockReportCount() > 0) {
        blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm registration: `hdfs dfsadmin -report`; the DN re-registers on its next heartbeat so this is usually self-healing — watch that subsequent BRs succeed
2. Restart the datanode to force immediate re-registration: `hdfs --daemon restart datanode`
3. Check dfs.hosts / dfs.hosts.exclude and refresh: `hdfs dfsadmin -refreshNodes` if the node was accidentally excluded
4. During NN restarts, expect these transiently until all DNs re-register; no data risk — reports are retried

Example fix

# before: full BR rejected during NN restart
# NN log: ProcessReport from dead or unregistered node: 10.0.0.5:9866

# after: verify/force registration, confirm next BR accepted
hdfs dfsadmin -refreshNodes
hdfs --daemon restart datanode
grep 'ProcessReport from dead' /var/log/hadoop-hdfs/*.log | tail  # should stop

try {
  nn.blockReport(bpRegistration, poolId, storage, reports, context);
} catch (IOException e) {
  if (e.getMessage().contains("dead or unregistered node")) {
    reRegisterWith Namenode(); // DN: handshake again, retry report on next cycle
  } else { throw e; }
}

When it happens

Trigger: Datanode sends its periodic full block report before (re-)registering — typical after NN restart/failover, when a DN's registration was evicted, or when the node was decommissioned but still running.

Common situations: NN restart in progress (safe mode) while DNs race to report; HA failover; datanode with stale registration talking to a new active; include/exclude list changes removing the node.

Related errors

• Datanode {datanode} not found.
• Cannot mark {blk} as corrupt because datanode {dn} ({datanod
• Cannot invalidate {b} because datanode {dn} does not exist.
• Data node {nodeID} is attempting to report storage ID {datan
• Got incremental block report from unregistered or dead node

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1bf26c8b1f15613b. Report an issue: GitHub.