apache/hadoop · error · IllegalArgumentException

Unexpected data node {} at an illegal network location

Error message

Unexpected data node {} at an illegal network location

What it means

In the node-group topology, add() resolves the new node's rack: it looks up (or creates) the InnerNodeWithNodeGroup at the node's location, then fetches that nodegroup's parent as the rack. The rack must be an InnerNode AND have a parent. A rack that is '/' (null parent) means the node's topology has only one layer, which the code explicitly labels a fault topology and rejects.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java:205

      // if node only with default rack info, here we need to add default 
      // nodegroup info
      if (NetworkTopology.DEFAULT_RACK.equals(node.getNetworkLocation())) {
        node.setNetworkLocation(node.getNetworkLocation() + 
            NetworkTopologyWithNodeGroup.DEFAULT_NODEGROUP);
      }
      Node nodeGroup = getNode(node.getNetworkLocation());
      if (nodeGroup == null) {
        nodeGroup = new InnerNodeWithNodeGroup(node.getNetworkLocation());
      }
      rack = getNode(nodeGroup.getNetworkLocation());

      // rack should be an innerNode and with parent.
      // note: rack's null parent case is: node's topology only has one layer, 
      //       so rack is recognized as "/" and no parent. 
      // This will be recognized as a node with fault topology.
      if (rack != null && 
          (!(rack instanceof InnerNode) || rack.getParent() == null)) {
        throw new IllegalArgumentException("Unexpected data node " 
            + node.toString() 
            + " at an illegal network location");
      }
      if (clusterMap.add(node)) {
        LOG.info("Adding a new node: " + NodeBase.getPath(node));
        if (rack == null) {
          // We only track rack number here
          incrementRacks();
        }
      }
      if(LOG.isDebugEnabled()) {
        LOG.debug("NetworkTopology became:\n" + this.toString());
      }
    } finally {
      netlock.writeLock().unlock();
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Map every host to a two-component location '/rack/nodegroup'
2. Return '/default-rack' for unmapped hosts so the class appends the default nodegroup itself
3. Verify the resulting tree (toString()/dfsadmin -printTopology): every leaf must sit at /rack/nodegroup/host

Example fix

# before (script output, one layer)
host1 -> /ng1

# after (two components)
host1 -> /rack1/ng1

String loc = NodeBase.normalize(node.getNetworkLocation());
int components = loc.equals("/") ? 0 : loc.substring(1).split("/").length;
if (components != 2) {
  throw new IllegalArgumentException(
      "Node-group topology needs /rack/nodegroup, got: " + loc);
}
topologyWithNodeGroup.add(node);

static boolean isNodeGroupLocation(String loc) {
  if (loc == null || loc.isEmpty() || loc.charAt(0) != '/') return false;
  return loc.substring(1).split("/").length == 2;
}

catch (IllegalArgumentException e) {
  if (e.getMessage().contains("illegal network location")) {
    // rewrite the node location to /rack/nodegroup and retry registration
  }
}

When it happens

Trigger: A node whose network location has a single component (e.g. '/ng1' or '/default-nodegroup'): the nodegroup's own location resolves to '/', so rack = the root clusterMap whose getParent() is null, and the IllegalArgumentException fires. The other branch fires when the rack path is occupied by a leaf.

Common situations: Topology scripts not updated for node-group awareness that return '/rack' or '/nodegroup' instead of '/rack/nodegroup'. Note '/default-rack' is safe: add() auto-appends DEFAULT_NODEGROUP to get '/default-rack/default-nodegroup'; any other single-level location is not.

Related errors

• Failed to add {}: You cannot have a rack and a non-rack node
• Unexpected data node {} at an illegal network location
• Not allow to add an inner node: {}
• Not allow to remove an inner node: {}
• ${node}is not an ancestor of ${n}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/656793a76ee0355e. Report an issue: GitHub.