apache/hadoop · warning · IOException

Cannot invalidate {b} because datanode {dn} does not exist.

Error message

Cannot invalidate {b} because datanode {dn} does not exist.

What it means

Thrown by BlockManager.invalidateBlock while the NameNode processes a reported corrupt replica and wants to schedule deletion on the owning datanode, but getDatanode(dn) no longer finds the node. Between counting replicas and queueing the invalidation the node disappeared from DatanodeManager (unregistered/dead/removed), so the invalidation cannot be queued and the corruption-processing RPC fails.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:2033

      // the block is over-replicated so invalidate the replicas immediately
      invalidateBlock(b, node, numberOfReplicas);
    } else if (isPopulatingReplQueues()) {
      // add the block to neededReconstruction
      updateNeededReconstructions(b.getStored(), -1, 0);
    }
  }

  /**
   * Invalidates the given block on the given datanode.
   * @return true if the block was successfully invalidated and no longer
   * present in the BlocksMap
   */
  private boolean invalidateBlock(BlockToMarkCorrupt b, DatanodeInfo dn,
      NumberReplicas nr) throws IOException {
    blockLog.debug("BLOCK* invalidateBlock: {} on {}", b, dn);
    DatanodeDescriptor node = getDatanodeManager().getDatanode(dn);
    if (node == null) {
      throw new IOException("Cannot invalidate " + b
          + " because datanode " + dn + " does not exist.");
    }

    // Check how many copies we have of the block
    if (nr.replicasOnStaleNodes() > 0 && !deleteCorruptReplicaImmediately) {
      if (blockLog.isDebugEnabled()) {
        blockLog.debug("BLOCK* invalidateBlocks: postponing " +
            "invalidation of {} on {} because {} replica(s) are located on " +
            "nodes with potentially out-of-date block reports", b, dn,
            nr.replicasOnStaleNodes());
      }
      postponeBlock(b.getCorrupted());
      return false;
    } else {
      // we already checked the number of replicas in the caller of this
      // function and know there are enough live replicas, so we can delete it.
      addToInvalidates(b.getCorrupted(), dn);
      removeStoredBlock(b.getStored(), node);

View on GitHub (pinned to 2add963021)

Solutions

Treat as transient: after the datanode re-registers, its next reports and replication will converge; run `hdfs fsck / -list-corruptfileblocks` to confirm no lingering corrupt files
Restart the datanode if it should remain in the cluster so it re-registers
If the node was removed on purpose, clear its stale entry (it will be re-registered or retired) and let the over-replication/corruption handlers clean up
If corrupt files persist, trigger replication of corrupt blocks via `hdfs fsck / -corruptfiles` review or -move/-delete as appropriate

Example fix

# before: invalidation races node removal
# NN log: Cannot invalidate blk_x on dn because datanode does not exist

# after: confirm node state and re-register or retire
hdfs dfsadmin -report | grep -B1 -A4 <host>
hdfs --daemon restart datanode   # if node stays
hdfs fsck / -list-corruptfileblocks   # verify cleanup converged

Defensive patterns

Strategy: retry

Try / catch

try {
  invalidateBlock(b, dn, nr);
} catch (IOException e) {
  if (e.getMessage().contains("datanode") && e.getMessage().contains("does not exist")) {
    // node left the cluster; invalidation moot — verify block converges via fsck
    LOG.debug("Skip invalidation, node gone: {}", dn);
  } else { throw e; }
}

Prevention

Treat as benign during node decommission/removal; verify with fsck afterwards
Avoid decommissioning and re-adding nodes during heavy corruption-report activity
Keep dfs.datanode.dead.interval sane so stale nodes re-register rather than vanish from maps
Re-run `hdfs fsck` to confirm no pending corrupt replicas remain

When it happens

Trigger: Corrupt replica report for node X arrives (often delayed/retried IBR or a queued corruption from a previous report); during handling, X unregisters (decommission finalize, NN restart) and invalidateBlock cannot resolve it. Common with postponed invalidations replayed after stale node heartbeats resume.

Common situations: Node decommissioned or unregistered between report and processing; NN failover where pending corruption states replay against a changed node set; long-stale node whose stale replicas were postponed (deleteCorruptReplicaImmediately=false) then it dies.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/10964a85a525370c. Report an issue: GitHub.