hashicorp/nomad · error

cannot find node %q for plugin instance %q

Error message

cannot find node %q for plugin instance %q

What it means

This error is produced by clientIDsForController when resolving which Nomad client node runs a CSI controller plugin. For a given controller plugin instance (identified by its allocation), the code looks up the node that hosts the client via getNodeForRpc; if the lookup fails or returns nil, it accumulates this error into a multi-error and skips the instance. It means the server's state snapshot has no node record matching the client that runs the controller plugin allocation.

Source

Thrown at nomad/client_csi_endpoint.go:334

	for clientID, controller := range plugin.Controllers {
		if !controller.IsController() {
			// we don't have separate types for CSIInfo depending on whether
			// it's a controller or node. this error should never make it to
			// production
			merr = errors.Join(merr, fmt.Errorf(
				"plugin instance %q is not a controller but was registered as one - this is always a bug", controller.AllocID))
			continue
		}

		if !controller.Healthy {
			merr = errors.Join(merr, fmt.Errorf(
				"plugin instance %q is not healthy", controller.AllocID))
			continue
		}

		node, err := getNodeForRpc(snap, clientID)
		if err != nil || node == nil {
			merr = errors.Join(merr, fmt.Errorf(
				"cannot find node %q for plugin instance %q", clientID, controller.AllocID))
			continue
		}

		if node.Status != structs.NodeStatusReady {
			merr = errors.Join(merr, fmt.Errorf(
				"node %q for plugin instance %q is not ready", clientID, controller.AllocID))
			continue
		}

		clientIDs = append(clientIDs, clientID)
	}

	if len(clientIDs) == 0 {
		return nil, fmt.Errorf("failed to find clients running controller plugin %q: %v",
			pluginID, merr)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Run `nomad node status` to verify the node running the controller plugin allocation is registered and healthy.
2. Check the plugin allocation (`nomad alloc status <alloc-id>`); if the allocation is stale, restart the CSI plugin job so it reschedules onto a live client.
3. Inspect server logs for errors from getNodeForRpc to see why the node lookup fails.
4. If the node was permanently removed, force GC or update the CSI plugin job to clear stale references.

Example fix

// Operator-level fix: reschedule the plugin on a live node
// before
$ nomad status <csi-plugin-job>   # alloc on dead node 'abc123'
// after
$ nomad job stop <csi-plugin-job> && nomad job run <csi-plugin-job>  # lands on a registered node

// before issuing a CSI controller RPC, verify the node is registered
nodes, _, _ := client.Nodes().List(nil)
known := map[string]bool{}
for _, n := range nodes { known[n.ID] = true }
// resolve the plugin alloc's node first; skip if !known[nodeID]

When it happens

Trigger: Calling a CSI controller RPC (via sendCSIControllerRPC, e.g. volume create/delete/attach) while the node running the controller plugin allocation cannot be resolved: the node record was deleted/garbage-collected from state, the client ID in the plugin's allocation is stale, or getNodeForRpc returns an error for that client.

Common situations: Deregistered or GC'd nodes still referenced by a CSI plugin allocation; stale state after a node left the cluster and its allocations were rescheduled; a corrupted/incomplete plugin fingerprint entry pointing at a nonexistent client.

Related errors

• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required
• CSI.NodeDetachVolume: PluginID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c07394d92ed3a5e0. Report an issue: GitHub.