{"record":{"id":"1b2c0dbb03203d38","repo":"apache/hadoop","slug":"got-incremental-block-report-from-unregistered-or","errorCode":null,"errorMessage":"Got incremental block report from unregistered or dead node","messagePattern":"Got incremental block report from unregistered or dead node","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java","lineNumber":4654,"sourceCode":"    }\n    return true;\n  }\n\n  /**\n   * The given node is reporting incremental information about some blocks.\n   * This includes blocks that are starting to be received, completed being\n   * received, or deleted.\n   * \n   * This method must be called with FSNamesystem lock held.\n   */\n  public void processIncrementalBlockReport(final DatanodeID nodeID,\n      final StorageReceivedDeletedBlocks srdb) throws IOException {\n    assert namesystem.hasWriteLock(RwLockMode.GLOBAL);\n    final DatanodeDescriptor node = datanodeManager.getDatanode(nodeID);\n    if (node == null || !node.isRegistered()) {\n      blockLog.warn(\"BLOCK* processIncrementalBlockReport\"\n              + \" is received from dead or unregistered node {}\", nodeID);\n      throw new IOException(\n          \"Got incremental block report from unregistered or dead node\");\n    }\n\n    boolean successful = false;\n    try {\n      processIncrementalBlockReport(node, srdb);\n      successful = true;\n    } finally {\n      if (!successful) {\n        node.setForceRegistration(true);\n      }\n    }\n  }\n\n  private void processIncrementalBlockReport(final DatanodeDescriptor node,\n      final StorageReceivedDeletedBlocks srdb) throws IOException {\n    DatanodeStorageInfo storageInfo =\n        node.getStorageInfo(srdb.getStorage().getStorageID());","sourceCodeStart":4636,"sourceCodeEnd":4672,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L4636-L4672","documentation":"Thrown by BlockManager.processIncrementalBlockReport when a datanode's incremental report (received/deleted blocks) arrives from a nodeID that is unknown or not registered. Incremental reports mutate live block maps, so the NN rejects them from unregistered senders; as a self-healing measure the catch/finally sets node.setForceRegistration(true) so the next heartbeat re-registers the node.","triggerScenarios":"DN sends IBR (e.g., receivedBlock acks during a pipeline write, or deleted-block acks) after its registration lapsed: NN restart/failover, node removed by admin, registration timeout; also a first IBR racing the registration RPC on DN startup.","commonSituations":"Writes failing right after NN failover with this error inside the pipeline heartbeat path; rolling upgrades; nodes under decommission still acking deletions.","solutions":["Mostly self-healing: forceRegistration is set, next heartbeat re-registers; monitor that the error stops after one or two heartbeat intervals","If it persists, check why registration is failing (include-list, storage ID conflicts, NN log 'registerDatanode' entries) and `hdfs dfsadmin -refreshNodes`","Restart the datanode to force a clean registration handshake","During planned NN restarts, expect a burst of these; they are benign if followed by successful registration"],"exampleFix":"# before: write pipeline fails during NN failover\n# NN log: Got incremental block report from unregistered or dead node\n\n# after: let heartbeat re-register (auto), verify with\nhdfs dfsadmin -report | grep -A3 <host>\n# if repeated: hdfs --daemon restart datanode","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  nn.blockReceivedAndDeleted(registration, poolId, srdb);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"unregistered or dead node\")) {\n    // NN set forceRegistration; next heartbeat re-registers — retry IBR after heartbeat\n    heartbeat(); retryBlockReceivedAndDeleted(srdb);\n  } else { throw e; }\n}","preventionTips":["Rely on built-in self-healing: forceRegistration triggers re-registration on heartbeat","During failovers, expect a short window of IBR rejections — build write retries accordingly","Keep DN heartbeat interval (dfs.heartbeat.interval) low on churn-heavy clusters","Verify with dfsadmin -report that the node eventually stays registered"],"tags":["hdfs","incremental-block-report","datanode-registration","namenode","failover"],"backgroundTag":"unregistered-datanode","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}