{"record":{"id":"10964a85a525370c","repo":"apache/hadoop","slug":"cannot-invalidate-b-because-datanode-dn-does-n","errorCode":null,"errorMessage":"Cannot invalidate {b} because datanode {dn} does not exist.","messagePattern":"Cannot invalidate (.+?) because datanode (.+?) does not exist\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java","lineNumber":2033,"sourceCode":"      // the block is over-replicated so invalidate the replicas immediately\n      invalidateBlock(b, node, numberOfReplicas);\n    } else if (isPopulatingReplQueues()) {\n      // add the block to neededReconstruction\n      updateNeededReconstructions(b.getStored(), -1, 0);\n    }\n  }\n\n  /**\n   * Invalidates the given block on the given datanode.\n   * @return true if the block was successfully invalidated and no longer\n   * present in the BlocksMap\n   */\n  private boolean invalidateBlock(BlockToMarkCorrupt b, DatanodeInfo dn,\n      NumberReplicas nr) throws IOException {\n    blockLog.debug(\"BLOCK* invalidateBlock: {} on {}\", b, dn);\n    DatanodeDescriptor node = getDatanodeManager().getDatanode(dn);\n    if (node == null) {\n      throw new IOException(\"Cannot invalidate \" + b\n          + \" because datanode \" + dn + \" does not exist.\");\n    }\n\n    // Check how many copies we have of the block\n    if (nr.replicasOnStaleNodes() > 0 && !deleteCorruptReplicaImmediately) {\n      if (blockLog.isDebugEnabled()) {\n        blockLog.debug(\"BLOCK* invalidateBlocks: postponing \" +\n            \"invalidation of {} on {} because {} replica(s) are located on \" +\n            \"nodes with potentially out-of-date block reports\", b, dn,\n            nr.replicasOnStaleNodes());\n      }\n      postponeBlock(b.getCorrupted());\n      return false;\n    } else {\n      // we already checked the number of replicas in the caller of this\n      // function and know there are enough live replicas, so we can delete it.\n      addToInvalidates(b.getCorrupted(), dn);\n      removeStoredBlock(b.getStored(), node);","sourceCodeStart":2015,"sourceCodeEnd":2051,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2015-L2051","documentation":"Thrown by BlockManager.invalidateBlock while the NameNode processes a reported corrupt replica and wants to schedule deletion on the owning datanode, but getDatanode(dn) no longer finds the node. Between counting replicas and queueing the invalidation the node disappeared from DatanodeManager (unregistered/dead/removed), so the invalidation cannot be queued and the corruption-processing RPC fails.","triggerScenarios":"Corrupt replica report for node X arrives (often delayed/retried IBR or a queued corruption from a previous report); during handling, X unregisters (decommission finalize, NN restart) and invalidateBlock cannot resolve it. Common with postponed invalidations replayed after stale node heartbeats resume.","commonSituations":"Node decommissioned or unregistered between report and processing; NN failover where pending corruption states replay against a changed node set; long-stale node whose stale replicas were postponed (deleteCorruptReplicaImmediately=false) then it dies.","solutions":["Treat as transient: after the datanode re-registers, its next reports and replication will converge; run `hdfs fsck / -list-corruptfileblocks` to confirm no lingering corrupt files","Restart the datanode if it should remain in the cluster so it re-registers","If the node was removed on purpose, clear its stale entry (it will be re-registered or retired) and let the over-replication/corruption handlers clean up","If corrupt files persist, trigger replication of corrupt blocks via `hdfs fsck / -corruptfiles` review or -move/-delete as appropriate"],"exampleFix":"# before: invalidation races node removal\n# NN log: Cannot invalidate blk_x on dn because datanode does not exist\n\n# after: confirm node state and re-register or retire\nhdfs dfsadmin -report | grep -B1 -A4 <host>\nhdfs --daemon restart datanode   # if node stays\nhdfs fsck / -list-corruptfileblocks   # verify cleanup converged","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  invalidateBlock(b, dn, nr);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"datanode\") && e.getMessage().contains(\"does not exist\")) {\n    // node left the cluster; invalidation moot — verify block converges via fsck\n    LOG.debug(\"Skip invalidation, node gone: {}\", dn);\n  } else { throw e; }\n}","preventionTips":["Treat as benign during node decommission/removal; verify with fsck afterwards","Avoid decommissioning and re-adding nodes during heavy corruption-report activity","Keep dfs.datanode.dead.interval sane so stale nodes re-register rather than vanish from maps","Re-run `hdfs fsck` to confirm no pending corrupt replicas remain"],"tags":["hdfs","invalidation","corrupt-replica","datanode-registration","namenode"],"backgroundTag":"unregistered-datanode","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}