apache/hadoop · error · InvalidTopologyException

Failed to add {}: You cannot have a rack and a non-rack node

Error message

Failed to add {}: You cannot have a rack and a non-rack node at the same level of the network topology.

What it means

add(Node) enforces a uniform leaf depth: the first inserted leaf records depthOfAllLeaves, and every later leaf whose depth (locationToDepth(location)+1) differs throws InvalidTopologyException. Hadoop's cluster map cannot represent a rack and a non-rack node at the same topology level, so mixed-depth mappings are rejected.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java:145

  /** Add a leaf node
   * Update node counter & rack counter if necessary
   * @param node node to be added; can be null
   * @exception IllegalArgumentException if add a node to a leave 
                                         or node to be added is not a leaf
   */
  public void add(Node node) {
    if (node==null) return;
    int newDepth = NodeBase.locationToDepth(node.getNetworkLocation()) + 1;
    netlock.writeLock().lock();
    try {
      if( node instanceof InnerNode ) {
        throw new IllegalArgumentException(
          "Not allow to add an inner node: "+NodeBase.getPath(node));
      }
      if ((depthOfAllLeaves != -1) && (depthOfAllLeaves != newDepth)) {
        LOG.error("Error: can't add leaf node {} at depth {} to topology:{}\n",
            NodeBase.getPath(node), newDepth, this);
        throw new InvalidTopologyException("Failed to add " + NodeBase.getPath(node) +
            ": You cannot have a rack and a non-rack node at the same " +
            "level of the network topology.");
      }
      Node rack = getNodeForNetworkLocation(node);
      if (rack != null && !(rack instanceof InnerNode)) {
        throw new IllegalArgumentException("Unexpected data node " 
                                           + node.toString() 
                                           + " at an illegal network location");
      }
      if (clusterMap.add(node)) {
        LOG.info("Adding a new node: "+NodeBase.getPath(node));
        if (rack == null) {
          incrementRacks();
        }
        interAddNodeWithEmptyRack(node);
        if (depthOfAllLeaves == -1) {
          depthOfAllLeaves = node.getLevel();
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Run `hdfs dfsadmin -printTopology` and identify which node reports a different depth than the rest; fix that host's mapping
2. Make the topology script or StaticMapping return the same number of path components for every host (all /rack or all /dc/rack)
3. Fix scripts that fail or time out for some hosts so they never silently emit /default-rack in a multi-level cluster

Example fix

# before (script output, inconsistent)
host1 -> /default-rack
host2 -> /dc1/rack1

# after (uniform depth)
host1 -> /dc1/rack1
host2 -> /dc1/rack2

int newDepth = NodeBase.locationToDepth(node.getNetworkLocation()) + 1;
for (Node existing : topology.getLeaves(NodeBase.ROOT)) {
  if (existing.getLevel() != newDepth) {
    throw new InvalidTopologyException("Depth mismatch for "
        + NodeBase.getPath(node));
  }
}
topology.add(node);

static boolean isUniformDepth(NetworkTopology t, Node candidate) {
  int d = NodeBase.locationToDepth(candidate.getNetworkLocation()) + 1;
  return t.getLeaves(NodeBase.ROOT).stream().allMatch(n -> n.getLevel() == d);
}

catch (InvalidTopologyException e) { // config problem, not transient: log and refuse to register the node
  LOG.error("Rejecting node with non-uniform topology depth: {}", e.getMessage());
}

When it happens

Trigger: The first node registers at /default-rack (unmapped fallback) and a later node at /dc1/rack1, or vice versa: any add() where the new node's network location has a different number of path components than the existing leaves. LOG.error prints the offending path, its depth, and the current topology just before the throw.

Common situations: net.topology.script.file-name returning different-depth outputs per host (script bug, DNS timeout for some hosts silently falling back to /default-rack); hosts missing from StaticMapping key-value data while others get multi-level racks; changing script output shape while the cluster is running.

Related errors

• Unexpected data node {} at an illegal network location
• Unexpected data node {} at an illegal network location
• Network Location is null
• Network Location path does not start with /: {}
• Configured cluster topology should be org.apache.hadoop.net.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef414a4d6968991c. Report an issue: GitHub.