{"record":{"id":"f3e65abfa29bb477","repo":"hashicorp/nomad","slug":"failed-to-find-clients-running-controller-plugin","errorCode":null,"errorMessage":"failed to find clients running controller plugin %q: %v","messagePattern":"failed to find clients running controller plugin %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":349,"sourceCode":"\n\t\tnode, err := getNodeForRpc(snap, clientID)\n\t\tif err != nil || node == nil {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"cannot find node %q for plugin instance %q\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif node.Status != structs.NodeStatusReady {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"node %q for plugin instance %q is not ready\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tclientIDs = append(clientIDs, clientID)\n\t}\n\n\tif len(clientIDs) == 0 {\n\t\treturn nil, fmt.Errorf(\"failed to find clients running controller plugin %q: %v\",\n\t\t\tpluginID, merr)\n\t}\n\n\t// Many plugins don't handle concurrent requests as described in the spec,\n\t// and have undocumented expectations of using k8s-specific sidecars to\n\t// leader elect. Sort the client IDs so that we prefer sending requests to\n\t// the same controller to hack around this.\n\tslices.Sort(clientIDs)\n\n\treturn clientIDs, nil\n}\n","sourceCodeStart":331,"sourceCodeEnd":361,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L331-L361","documentation":"This is the terminal error returned by clientIDsForController when, after examining all plugin instances for the requested CSI controller plugin, zero healthy client IDs remain. It wraps the aggregated per-instance errors (node-not-found / node-not-ready from errors 2100/2101), so the %v carries the root causes. The controller RPC is not attempted at all.","triggerScenarios":"Calling any CSI controller operation (e.g. nomad volume create/delete/detach, snapshot ops) when the controller plugin job has no allocations on ready nodes: all plugin allocations failed, the plugin job is stopped, or every hosting node is down/unresolvable.","commonSituations":"CSI controller plugin job crashed or was stopped; plugin deployed only on clients that are now down; plugin ID typo/mismatch between volume spec's controller plugin and the job's csi_plugin id; plugin not yet deployed to the cluster.","solutions":["Read the wrapped %v detail to see whether instances failed node lookup or node-not-ready, then fix accordingly.","Verify the controller plugin is running: `nomad plugin status <plugin-id>` and `nomad job status <csi-plugin-job>`.","Confirm the plugin ID in your volume specification matches the controller plugin ID in the job's csi_plugin block.","Deploy/restart the CSI plugin job so at least one controller instance runs on a ready node."],"exampleFix":"// before (volume spec references wrong plugin)\ncontroller_required = true\nplugin_id = \"aws-ebs-controller\"\n// after (matches job csi_plugin.id)\nplugin_id = \"ebs.csi.aws.com\"","handlingStrategy":"validation","validationCode":"plugin, _, err := client.CSIPlugins().Info(pluginID, nil)\nif err != nil || plugin == nil {\n    return fmt.Errorf(\"controller plugin %s not deployed\", pluginID)\n}\nif plugin.ControllersHealthy < 1 {\n    return fmt.Errorf(\"controller plugin %s has 0 healthy controllers\", pluginID)\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always run CSI controller plugins as a job with at least one running alloc on a ready node.","Keep plugin IDs in volume specs identical to job csi_plugin.id.","Alert on plugin.ControllersHealthy == 0 via Nomad API monitoring."],"tags":["csi","storage","nomad","plugin-missing"],"backgroundTag":"csi-controller-plugin-unavailable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}