apache/hadoop · warning · HadoopIllegalArgumentException

Datanode {datanode} not found.

Error message

Datanode {datanode} not found.

What it means

Thrown as HadoopIllegalArgumentException from BlockManager.getBlocksWithLocations when a datanode (typically the Balancer's datanode-side Mover/Balancer agent) requests the list of that node's blocks but the NameNode has no DatanodeDescriptor registered for the given DatanodeID. It means the node is unknown to the NN — it never registered, was decommissioned and removed, or its registration was evicted.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:1760

    }
    if(iFile.isUnderConstruction()) {
      return true;
    }
    if (iFile.getAccessTime() > time || iFile.getModificationTime() > time) {
      return true;
    }
    return false;
  }

  /** Get all blocks with location information from a datanode. */
  public BlocksWithLocations getBlocksWithLocations(final DatanodeID datanode,
      final long size, final long minBlockSize, final long timeInterval,
      final StorageType storageType) throws UnregisteredNodeException {
    final DatanodeDescriptor node = getDatanodeManager().getDatanode(datanode);
    if (node == null) {
      blockLog.warn("BLOCK* getBlocks: Asking for blocks from an" +
          " unrecorded node {}", datanode);
      throw new HadoopIllegalArgumentException(
          "Datanode " + datanode + " not found.");
    }

    int numBlocks = node.numBlocks();
    if(numBlocks == 0) {
      return new BlocksWithLocations(new BlockWithLocations[0]);
    }

    // skip stale storage, then choose specific storage type.
    DatanodeStorageInfo[] storageInfos = Arrays
        .stream(node.getStorageInfos())
        .filter(s -> !s.areBlockContentsStale())
        .filter(s -> storageType == null || s.getStorageType().equals(storageType))
        .toArray(DatanodeStorageInfo[]::new);

    // starting from a random block
    int startBlock = ThreadLocalRandom.current().nextInt(numBlocks);
    Iterator<BlockInfo> iter = node.getBlockIterator(startBlock, storageInfos);

View on GitHub (pinned to 2add963021)

Solutions

1. Restart the balancer after datanodes have (re)registered with the active NameNode — verify with `hdfs dfsadmin -report | grep -A2 Datanode`
2. Point the balancer at the active NN of the correct namespace (-fs hdfs://correct-nameservice)
3. If NN just restarted, wait for block reports/registrations to complete and exit safe mode, then rerun
4. Remove/retire the stale datanode reference if the node was decommissioned intentionally

Example fix

# before: balancer runs through NN restart / failover
hdfs --daemon start balancer  # getBlocks -> Datanode ... not found

# after: confirm registrations, then start balancer against active NN
hdfs dfsadmin -report | grep -c '^Datanode'
hdfs --daemon start balancer -fs hdfs://mycluster   # nameservice of active NN

// Balancer/tool side: confirm the datanode is known to the NN before getBlocks
DatanodeInfo[] live = ((DistributedFileSystem) fs).getDataNodeStats(DatanodeReportTypes.LIVE);
boolean known = Arrays.stream(live).anyMatch(d -> d.getXferAddr().equals(node.getXferAddr()));
if (known) { nn.getBlockLocations(node, size, minSize, interval, type); }

try {
  blocks = nn.getBlockLocations(node, size, minSize, interval, type);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("not found")) { reRegisterOrSkip(node); /* restart DN or skip node */ }
  else { throw e; }
}

When it happens

Trigger: Balancer/Datanode calls DataNodeProtocol getBlocks(datanodeID, size, ...) after the NN restarted and lost (or reordered) registration, or for a node that deregistered/decommissioned; NN failover to a standby that has not seen this DN register yet; stale balancer instance pointing at a different cluster/namespace.

Common situations: Balancer started while NameNode is in startup/safe mode before datanodes re-registered; HA failover with running balancer; running the balancer against the wrong NameNode/namespace (cluster mismatch); node decommissioned mid-balancer-run.

Related errors

• Unsupported protocol found when creating the proxy connectio
• Cannot mark {blk} as corrupt because datanode {dn} ({datanod
• Cannot invalidate {b} because datanode {dn} does not exist.
• ProcessReport from dead or unregistered node: {nodeID}
• Got incremental block report from unregistered or dead node

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/feffa08767015b22. Report an issue: GitHub.