{"record":{"id":"05318a253f3dbdec","repo":"apache/hadoop","slug":"cannot-mark-blk-as-corrupt-because-datanode-dn","errorCode":null,"errorMessage":"Cannot mark {blk} as corrupt because datanode {dn} ({datanodeUuid}) does not exist","messagePattern":"Cannot mark (.+?) as corrupt because datanode (.+?) \\((.+?)\\) does not exist","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java","lineNumber":1923,"sourceCode":"   * for logging purposes\n   */\n  public void findAndMarkBlockAsCorrupt(final ExtendedBlock blk,\n      final DatanodeInfo dn, String storageID, String reason) throws IOException {\n    assert namesystem.hasWriteLock(RwLockMode.BM);\n    final Block reportedBlock = blk.getLocalBlock();\n    final BlockInfo storedBlock = getStoredBlock(reportedBlock);\n    if (storedBlock == null) {\n      // Check if the replica is in the blockMap, if not\n      // ignore the request for now. This could happen when BlockScanner\n      // thread of Datanode reports bad block before Block reports are sent\n      // by the Datanode on startup\n      blockLog.debug(\"BLOCK* findAndMarkBlockAsCorrupt: {} not found\", blk);\n      return;\n    }\n\n    DatanodeDescriptor node = getDatanodeManager().getDatanode(dn);\n    if (node == null) {\n      throw new IOException(\"Cannot mark \" + blk\n          + \" as corrupt because datanode \" + dn + \" (\" + dn.getDatanodeUuid()\n          + \") does not exist\");\n    }\n    DatanodeStorageInfo storage = null;\n    if (storageID != null) {\n      storage = node.getStorageInfo(storageID);\n    }\n    if (storage == null) {\n      storage = storedBlock.findStorageInfo(node);\n    }\n\n    if (storage == null) {\n      blockLog.debug(\"BLOCK* findAndMarkBlockAsCorrupt: {} not found on {}\", blk, dn);\n      return;\n    }\n    markBlockAsCorrupt(new BlockToMarkCorrupt(reportedBlock, storedBlock,\n            blk.getGenerationStamp(), reason, Reason.CORRUPTION_REPORTED),\n        storage, node);","sourceCodeStart":1905,"sourceCodeEnd":1941,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L1905-L1941","documentation":"Thrown by BlockManager.findAndMarkBlockAsCorrupt when a datanode reports a corrupt replica (via incremental block report / block scanner) but datanodeManager.getDatanode(dn) returns null — the NameNode has no registered DatanodeDescriptor for the reporting node. The NN cannot record corruption against a node it does not know, so it fails the report.","triggerScenarios":"Datanode sends received/deleted/corrupt IBR (processIncrementalBlockReport -> tryToInvalidate / markBlockAsCorrupt path) after it was removed from DatanodeManager (unregistered, decommission finalized) or before registration completed; NN restarted and DN reports before its storage re-registers.","commonSituations":"NN restart/failover racing datanode incremental reports; decommissioned or dead-then-rejoining node whose registration was evicted; rolling upgrade where DNs report on a fresh RPC to a standby-turned-active.","solutions":["Verify datanode registration state: `hdfs dfsadmin -report` and NN log for 'Datanode denied/unregistered' entries","Restart the affected datanode (`hdfs --daemon restart datanode`) so it re-registers before reporting","If NN just restarted, wait for registration + full block reports; transient reports during this window are dropped safely","Check dfs.hosts/dfs.hosts.exclude if the node was deliberately removed — this is then expected noise, retire the DN"],"exampleFix":"# before: DN reports corrupt replica while unregistered (NN restarted)\n# NN log: Cannot mark blk_x as corrupt because datanode ... does not exist\n\n# after: re-register then let block scanner re-report\nhdfs --daemon restart datanode\n# after 'RegisterDatanode' in NN log, rescan reports succeed:\nhdfs dfsadmin -fs hdfs://nn -report | grep -A3 <host>","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// NN-side handler in tests/tools wrapping markBlockAsCorrupt paths:\ntry {\n  bm.findAndMarkBlockAsCorrupt(reportedBlock, dn, storageID, reason);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"does not exist\")) { LOG.debug(\"DN {} unregistered; skip corruption report\", dn); }\n  else { throw e; }\n}","preventionTips":["Ensure datanodes register before enabling block scanner reporting (post-restart ordering)","Monitor NN log for registration churn (unregister/register cycles)","Run `hdfs fsck -list-corruptfileblocks` after NN restarts to confirm corruption state converged","Keep include/exclude lists stable during corruption handling windows"],"tags":["hdfs","corrupt-replica","datanode-registration","incremental-block-report","namenode"],"backgroundTag":"unregistered-datanode","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}