apache/hadoop · warning · IOException

Cannot mark {blk} as corrupt because datanode {dn} ({datanod

Error message

Cannot mark {blk} as corrupt because datanode {dn} ({datanodeUuid}) does not exist

What it means

Thrown by BlockManager.findAndMarkBlockAsCorrupt when a datanode reports a corrupt replica (via incremental block report / block scanner) but datanodeManager.getDatanode(dn) returns null — the NameNode has no registered DatanodeDescriptor for the reporting node. The NN cannot record corruption against a node it does not know, so it fails the report.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:1923

   * for logging purposes
   */
  public void findAndMarkBlockAsCorrupt(final ExtendedBlock blk,
      final DatanodeInfo dn, String storageID, String reason) throws IOException {
    assert namesystem.hasWriteLock(RwLockMode.BM);
    final Block reportedBlock = blk.getLocalBlock();
    final BlockInfo storedBlock = getStoredBlock(reportedBlock);
    if (storedBlock == null) {
      // Check if the replica is in the blockMap, if not
      // ignore the request for now. This could happen when BlockScanner
      // thread of Datanode reports bad block before Block reports are sent
      // by the Datanode on startup
      blockLog.debug("BLOCK* findAndMarkBlockAsCorrupt: {} not found", blk);
      return;
    }

    DatanodeDescriptor node = getDatanodeManager().getDatanode(dn);
    if (node == null) {
      throw new IOException("Cannot mark " + blk
          + " as corrupt because datanode " + dn + " (" + dn.getDatanodeUuid()
          + ") does not exist");
    }
    DatanodeStorageInfo storage = null;
    if (storageID != null) {
      storage = node.getStorageInfo(storageID);
    }
    if (storage == null) {
      storage = storedBlock.findStorageInfo(node);
    }

    if (storage == null) {
      blockLog.debug("BLOCK* findAndMarkBlockAsCorrupt: {} not found on {}", blk, dn);
      return;
    }
    markBlockAsCorrupt(new BlockToMarkCorrupt(reportedBlock, storedBlock,
            blk.getGenerationStamp(), reason, Reason.CORRUPTION_REPORTED),
        storage, node);

View on GitHub (pinned to 2add963021)

Solutions

1. Verify datanode registration state: `hdfs dfsadmin -report` and NN log for 'Datanode denied/unregistered' entries
2. Restart the affected datanode (`hdfs --daemon restart datanode`) so it re-registers before reporting
3. If NN just restarted, wait for registration + full block reports; transient reports during this window are dropped safely
4. Check dfs.hosts/dfs.hosts.exclude if the node was deliberately removed — this is then expected noise, retire the DN

Example fix

# before: DN reports corrupt replica while unregistered (NN restarted)
# NN log: Cannot mark blk_x as corrupt because datanode ... does not exist

# after: re-register then let block scanner re-report
hdfs --daemon restart datanode
# after 'RegisterDatanode' in NN log, rescan reports succeed:
hdfs dfsadmin -fs hdfs://nn -report | grep -A3 <host>

// NN-side handler in tests/tools wrapping markBlockAsCorrupt paths:
try {
  bm.findAndMarkBlockAsCorrupt(reportedBlock, dn, storageID, reason);
} catch (IOException e) {
  if (e.getMessage().contains("does not exist")) { LOG.debug("DN {} unregistered; skip corruption report", dn); }
  else { throw e; }
}

When it happens

Trigger: Datanode sends received/deleted/corrupt IBR (processIncrementalBlockReport -> tryToInvalidate / markBlockAsCorrupt path) after it was removed from DatanodeManager (unregistered, decommission finalized) or before registration completed; NN restarted and DN reports before its storage re-registers.

Common situations: NN restart/failover racing datanode incremental reports; decommissioned or dead-then-rejoining node whose registration was evicted; rolling upgrade where DNs report on a fresh RPC to a standby-turned-active.

Related errors

• Cannot invalidate {b} because datanode {dn} does not exist.
• Got incremental block report from unregistered or dead node
• Datanode {datanode} not found.
• ProcessReport from dead or unregistered node: {nodeID}
• Datanode denied communication with namenode because hostname

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/05318a253f3dbdec. Report an issue: GitHub.