{"record":{"id":"c07394d92ed3a5e0","repo":"hashicorp/nomad","slug":"cannot-find-node-q-for-plugin-instance-q","errorCode":null,"errorMessage":"cannot find node %q for plugin instance %q","messagePattern":"cannot find node %q for plugin instance %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":334,"sourceCode":"\tfor clientID, controller := range plugin.Controllers {\n\t\tif !controller.IsController() {\n\t\t\t// we don't have separate types for CSIInfo depending on whether\n\t\t\t// it's a controller or node. this error should never make it to\n\t\t\t// production\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"plugin instance %q is not a controller but was registered as one - this is always a bug\", controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif !controller.Healthy {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"plugin instance %q is not healthy\", controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tnode, err := getNodeForRpc(snap, clientID)\n\t\tif err != nil || node == nil {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"cannot find node %q for plugin instance %q\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif node.Status != structs.NodeStatusReady {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"node %q for plugin instance %q is not ready\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tclientIDs = append(clientIDs, clientID)\n\t}\n\n\tif len(clientIDs) == 0 {\n\t\treturn nil, fmt.Errorf(\"failed to find clients running controller plugin %q: %v\",\n\t\t\tpluginID, merr)\n\t}\n","sourceCodeStart":316,"sourceCodeEnd":352,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L316-L352","documentation":"This error is produced by clientIDsForController when resolving which Nomad client node runs a CSI controller plugin. For a given controller plugin instance (identified by its allocation), the code looks up the node that hosts the client via getNodeForRpc; if the lookup fails or returns nil, it accumulates this error into a multi-error and skips the instance. It means the server's state snapshot has no node record matching the client that runs the controller plugin allocation.","triggerScenarios":"Calling a CSI controller RPC (via sendCSIControllerRPC, e.g. volume create/delete/attach) while the node running the controller plugin allocation cannot be resolved: the node record was deleted/garbage-collected from state, the client ID in the plugin's allocation is stale, or getNodeForRpc returns an error for that client.","commonSituations":"Deregistered or GC'd nodes still referenced by a CSI plugin allocation; stale state after a node left the cluster and its allocations were rescheduled; a corrupted/incomplete plugin fingerprint entry pointing at a nonexistent client.","solutions":["Run `nomad node status` to verify the node running the controller plugin allocation is registered and healthy.","Check the plugin allocation (`nomad alloc status <alloc-id>`); if the allocation is stale, restart the CSI plugin job so it reschedules onto a live client.","Inspect server logs for errors from getNodeForRpc to see why the node lookup fails.","If the node was permanently removed, force GC or update the CSI plugin job to clear stale references."],"exampleFix":"// Operator-level fix: reschedule the plugin on a live node\n// before\n$ nomad status <csi-plugin-job>   # alloc on dead node 'abc123'\n// after\n$ nomad job stop <csi-plugin-job> && nomad job run <csi-plugin-job>  # lands on a registered node","handlingStrategy":"validation","validationCode":"// before issuing a CSI controller RPC, verify the node is registered\nnodes, _, _ := client.Nodes().List(nil)\nknown := map[string]bool{}\nfor _, n := range nodes { known[n.ID] = true }\n// resolve the plugin alloc's node first; skip if !known[nodeID]","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check `nomad plugin status` shows controllers healthy before volume operations.","Avoid GC-ing nodes that still run CSI plugin allocations.","Monitor node deregistration events and alert when CSI plugin allocs lose their node."],"tags":["csi","storage","nomad","node-lookup"],"backgroundTag":"csi-controller-node-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}