hashicorp/nomad · error
node for plugin instance is not ready
Error message
node %q for plugin instance %q is not ready
What it means
clientIDsForController throws this when the node running a CSI controller plugin instance is found but its status is not "ready". The error is accumulated per instance into a multi-error; the RPC proceeds only if at least one healthy client remains. It indicates the client node hosting the controller plugin is down, draining, or ineligible to serve RPCs.
Solutions
- Check `nomad node status <node-id>`; bring the node back (restart nomad agent, fix network) so it reports ready.
- If the node is intentionally down, reschedule the CSI plugin job to another eligible client: `nomad job eval` or update the job's datacenter/constraints.
- If draining, either finish maintenance and undrain (`nomad node eligibility -enable <node>`) or move the plugin.
- Verify node heartbeats/TTL: fix clock skew or connectivity that causes the server to mark the node down.
Example fix
// before $ nomad node status n1 # Status: down // after $ sudo systemctl restart nomad && nomad node status n1 # Status: ready
Defensive patterns
Strategy: validation
Validate before calling
node, _, err := client.Nodes().Info(nodeID, nil)
if err != nil || node == nil || node.Status != "ready" {
return fmt.Errorf("node %s not ready for CSI controller RPC", nodeID)
} Prevention
- Gate automation on node.Status == "ready" before volume operations.
- Drain nodes with CSI plugin jobs only after rescheduling the plugin.
- Fix heartbeat/clock issues that spuriously mark nodes down.
When it happens
Trigger: Any CSI controller RPC (volume create/delete/snapshot/attach through sendCSIControllerRPC) where the node running the controller plugin allocation has a node.Status other than structs.NodeStatusReady (e.g. down, initializing, draining).
Common situations: Node crashed or agent stopped while its CSI plugin allocation still exists; node in draining state for maintenance; node failing heartbeats and marked down by servers; newly joined node not yet fully ready.
Related errors
- cannot change attachment mode of claimed volume
- cannot find node for plugin instance
- Claim.AllocationID is required
- Claim is required
- claiming volumes
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/7d30680aaf573164.
Report an issue: GitHub.
Appendix: source
Thrown at nomad/client_csi_endpoint.go:340
"plugin instance %q is not a controller but was registered as one - this is always a bug", controller.AllocID))
continue
}
if !controller.Healthy {
merr = errors.Join(merr, fmt.Errorf(
"plugin instance %q is not healthy", controller.AllocID))
continue
}
node, err := getNodeForRpc(snap, clientID)
if err != nil || node == nil {
merr = errors.Join(merr, fmt.Errorf(
"cannot find node %q for plugin instance %q", clientID, controller.AllocID))
continue
}
if node.Status != structs.NodeStatusReady {
merr = errors.Join(merr, fmt.Errorf(
"node %q for plugin instance %q is not ready", clientID, controller.AllocID))
continue
}
clientIDs = append(clientIDs, clientID)
}
if len(clientIDs) == 0 {
return nil, fmt.Errorf("failed to find clients running controller plugin %q: %v",
pluginID, merr)
}
// Many plugins don't handle concurrent requests as described in the spec,
// and have undocumented expectations of using k8s-specific sidecars to
// leader elect. Sort the client IDs so that we prefer sending requests to
// the same controller to hack around this.
slices.Sort(clientIDs)
View on GitHub (pinned to 482b49bf1a)