{"record":{"id":"7d30680aaf573164","repo":"hashicorp/nomad","slug":"node-q-for-plugin-instance-q-is-not-ready","errorCode":null,"errorMessage":"node %q for plugin instance %q is not ready","messagePattern":"node %q for plugin instance %q is not ready","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":340,"sourceCode":"\t\t\t\t\"plugin instance %q is not a controller but was registered as one - this is always a bug\", controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif !controller.Healthy {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"plugin instance %q is not healthy\", controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tnode, err := getNodeForRpc(snap, clientID)\n\t\tif err != nil || node == nil {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"cannot find node %q for plugin instance %q\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif node.Status != structs.NodeStatusReady {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"node %q for plugin instance %q is not ready\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tclientIDs = append(clientIDs, clientID)\n\t}\n\n\tif len(clientIDs) == 0 {\n\t\treturn nil, fmt.Errorf(\"failed to find clients running controller plugin %q: %v\",\n\t\t\tpluginID, merr)\n\t}\n\n\t// Many plugins don't handle concurrent requests as described in the spec,\n\t// and have undocumented expectations of using k8s-specific sidecars to\n\t// leader elect. Sort the client IDs so that we prefer sending requests to\n\t// the same controller to hack around this.\n\tslices.Sort(clientIDs)\n","sourceCodeStart":322,"sourceCodeEnd":358,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L322-L358","documentation":"clientIDsForController throws this when the node running a CSI controller plugin instance is found but its status is not \"ready\". The error is accumulated per instance into a multi-error; the RPC proceeds only if at least one healthy client remains. It indicates the client node hosting the controller plugin is down, draining, or ineligible to serve RPCs.","triggerScenarios":"Any CSI controller RPC (volume create/delete/snapshot/attach through sendCSIControllerRPC) where the node running the controller plugin allocation has a node.Status other than structs.NodeStatusReady (e.g. down, initializing, draining).","commonSituations":"Node crashed or agent stopped while its CSI plugin allocation still exists; node in draining state for maintenance; node failing heartbeats and marked down by servers; newly joined node not yet fully ready.","solutions":["Check `nomad node status <node-id>`; bring the node back (restart nomad agent, fix network) so it reports ready.","If the node is intentionally down, reschedule the CSI plugin job to another eligible client: `nomad job eval` or update the job's datacenter/constraints.","If draining, either finish maintenance and undrain (`nomad node eligibility -enable <node>`) or move the plugin.","Verify node heartbeats/TTL: fix clock skew or connectivity that causes the server to mark the node down."],"exampleFix":"// before\n$ nomad node status n1   # Status: down\n// after\n$ sudo systemctl restart nomad && nomad node status n1   # Status: ready","handlingStrategy":"validation","validationCode":"node, _, err := client.Nodes().Info(nodeID, nil)\nif err != nil || node == nil || node.Status != \"ready\" {\n    return fmt.Errorf(\"node %s not ready for CSI controller RPC\", nodeID)\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Gate automation on node.Status == \"ready\" before volume operations.","Drain nodes with CSI plugin jobs only after rescheduling the plugin.","Fix heartbeat/clock issues that spuriously mark nodes down."],"tags":["csi","storage","nomad","node-health"],"backgroundTag":"csi-controller-node-not-ready","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}