hashicorp/nomad · error

topology: no node distance

Error message

topology: no node distance

What it means

Topology.NodeDistance looks up the SLIT-style distance cost between two NUMA nodes by scanning st.Distances. If the target node ID is not found among the recorded distance entries, the invariant 'every node has a distance to every other node' is broken and the code panics with this message.

Source

Thrown at client/lib/numalib/topology.go:160

func (st *Topology) NodeDistance(node hw.NodeID, core Core) Cost {
	// todo(shoenig) we should memoize these values - they never change but
	// they get used a lot in numa scheduling

	// fast path, core is on node
	if core.NodeID == node {
		// return the distance to itself (100%)
		return st.Distances.cost(node, node)
	}

	// find a core on node to compare with
	for _, target := range st.Cores {
		if target.NodeID == node {
			return st.Distances.cost(target.NodeID, core.NodeID)
		}
	}

	// should not be possible
	panic("topology: no node distance")
}

// SupportsNUMA returns whether Nomad supports NUMA detection on the client's
// operating system. Currently only supported on Linux.
func (st *Topology) SupportsNUMA() bool {
	switch runtime.GOOS {
	case "linux":
		return true
	default:
		return false
	}
}

// GetNodes returns the set of NUMA Node IDs.
func (st *Topology) GetNodes() *idset.Set[hw.NodeID] {
	if st.nodeIDs.Empty() {
		st.nodeIDs = idset.From[hw.NodeID](st.Nodes)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Call SupportsNUMA() and verify the topology was fully detected before invoking NodeDistance
2. Ensure the client's NUMA detection (numalib) ran on the host and populated Distances (check /sys/devices/system/node/slit or ACPI SLIT availability)
3. Guard call sites with a check that the requested node ID exists in the topology's node list
4. Report a bug with the host's NUMA topology — an internally inconsistent Topology is a detection bug

Example fix

// before
cost := topo.NodeDistance(nodeID)
// after
if topo.SupportsNUMA() && nodeID < len(topo.Nodes) {
    cost = topo.NodeDistance(nodeID)
} else {
    cost = defaultLocalCost
}

func nodeExists(t *numalib.Topology, node uint64) bool {
    for _, n := range t.GetDistances() { if n.NodeID == node { return true } }
    return false
}

func safeNodeDistance(t *numalib.Topology, node int) (cost int, err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("no node distance for %d", node)
        }
    }()
    return t.NodeDistance(node), nil
}

When it happens

Trigger: Calling NodeDistance(node) where node is not present in the topology's Distances table — e.g. a core whose NodeID was not recorded during NUMA topology detection, or querying a node ID greater than the detected node count.

Common situations: Partial or failed NUMA detection on the host (no SLIT table, VM/container masking NUMA info) leaving an empty or incomplete Distances slice; calling NodeDistance before the topology has been fully populated; hardware/BIOS exposing cores mapped to an undeclared node.

Related errors

• numa scheduling requires Nomad Enterprise
• volume topology request update was not compatible with exist
• numa affinity must be one of none, prefer, or require
• invalid acl policy: %v
• failed to detect memset: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c1b6eeec8a510497. Report an issue: GitHub.