hashicorp/nomad · error

node %q for plugin instance %q is not ready

Error message

node %q for plugin instance %q is not ready

What it means

clientIDsForController throws this when the node running a CSI controller plugin instance is found but its status is not "ready". The error is accumulated per instance into a multi-error; the RPC proceeds only if at least one healthy client remains. It indicates the client node hosting the controller plugin is down, draining, or ineligible to serve RPCs.

Source

Thrown at nomad/client_csi_endpoint.go:340

				"plugin instance %q is not a controller but was registered as one - this is always a bug", controller.AllocID))
			continue
		}

		if !controller.Healthy {
			merr = errors.Join(merr, fmt.Errorf(
				"plugin instance %q is not healthy", controller.AllocID))
			continue
		}

		node, err := getNodeForRpc(snap, clientID)
		if err != nil || node == nil {
			merr = errors.Join(merr, fmt.Errorf(
				"cannot find node %q for plugin instance %q", clientID, controller.AllocID))
			continue
		}

		if node.Status != structs.NodeStatusReady {
			merr = errors.Join(merr, fmt.Errorf(
				"node %q for plugin instance %q is not ready", clientID, controller.AllocID))
			continue
		}

		clientIDs = append(clientIDs, clientID)
	}

	if len(clientIDs) == 0 {
		return nil, fmt.Errorf("failed to find clients running controller plugin %q: %v",
			pluginID, merr)
	}

	// Many plugins don't handle concurrent requests as described in the spec,
	// and have undocumented expectations of using k8s-specific sidecars to
	// leader elect. Sort the client IDs so that we prefer sending requests to
	// the same controller to hack around this.
	slices.Sort(clientIDs)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check `nomad node status <node-id>`; bring the node back (restart nomad agent, fix network) so it reports ready.
2. If the node is intentionally down, reschedule the CSI plugin job to another eligible client: `nomad job eval` or update the job's datacenter/constraints.
3. If draining, either finish maintenance and undrain (`nomad node eligibility -enable <node>`) or move the plugin.
4. Verify node heartbeats/TTL: fix clock skew or connectivity that causes the server to mark the node down.

Example fix

// before
$ nomad node status n1   # Status: down
// after
$ sudo systemctl restart nomad && nomad node status n1   # Status: ready

node, _, err := client.Nodes().Info(nodeID, nil)
if err != nil || node == nil || node.Status != "ready" {
    return fmt.Errorf("node %s not ready for CSI controller RPC", nodeID)
}

When it happens

Trigger: Any CSI controller RPC (volume create/delete/snapshot/attach through sendCSIControllerRPC) where the node running the controller plugin allocation has a node.Status other than structs.NodeStatusReady (e.g. down, initializing, draining).

Common situations: Node crashed or agent stopped while its CSI plugin allocation still exists; node in draining state for maintenance; node failing heartbeats and marked down by servers; newly joined node not yet fully ready.

Related errors

• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required
• CSI.NodeDetachVolume: PluginID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7d30680aaf573164. Report an issue: GitHub.