hashicorp/nomad · error
failed to find clients running controller plugin %q: %v
Error message
failed to find clients running controller plugin %q: %v
What it means
This is the terminal error returned by clientIDsForController when, after examining all plugin instances for the requested CSI controller plugin, zero healthy client IDs remain. It wraps the aggregated per-instance errors (node-not-found / node-not-ready from errors 2100/2101), so the %v carries the root causes. The controller RPC is not attempted at all.
Source
Thrown at nomad/client_csi_endpoint.go:349
node, err := getNodeForRpc(snap, clientID)
if err != nil || node == nil {
merr = errors.Join(merr, fmt.Errorf(
"cannot find node %q for plugin instance %q", clientID, controller.AllocID))
continue
}
if node.Status != structs.NodeStatusReady {
merr = errors.Join(merr, fmt.Errorf(
"node %q for plugin instance %q is not ready", clientID, controller.AllocID))
continue
}
clientIDs = append(clientIDs, clientID)
}
if len(clientIDs) == 0 {
return nil, fmt.Errorf("failed to find clients running controller plugin %q: %v",
pluginID, merr)
}
// Many plugins don't handle concurrent requests as described in the spec,
// and have undocumented expectations of using k8s-specific sidecars to
// leader elect. Sort the client IDs so that we prefer sending requests to
// the same controller to hack around this.
slices.Sort(clientIDs)
return clientIDs, nil
}
View on GitHub (pinned to 482b49bf1a)
Solutions
- Read the wrapped %v detail to see whether instances failed node lookup or node-not-ready, then fix accordingly.
- Verify the controller plugin is running: `nomad plugin status <plugin-id>` and `nomad job status <csi-plugin-job>`.
- Confirm the plugin ID in your volume specification matches the controller plugin ID in the job's csi_plugin block.
- Deploy/restart the CSI plugin job so at least one controller instance runs on a ready node.
Example fix
// before (volume spec references wrong plugin) controller_required = true plugin_id = "aws-ebs-controller" // after (matches job csi_plugin.id) plugin_id = "ebs.csi.aws.com"
Defensive patterns
Strategy: validation
Validate before calling
plugin, _, err := client.CSIPlugins().Info(pluginID, nil)
if err != nil || plugin == nil {
return fmt.Errorf("controller plugin %s not deployed", pluginID)
}
if plugin.ControllersHealthy < 1 {
return fmt.Errorf("controller plugin %s has 0 healthy controllers", pluginID)
} Prevention
- Always run CSI controller plugins as a job with at least one running alloc on a ready node.
- Keep plugin IDs in volume specs identical to job csi_plugin.id.
- Alert on plugin.ControllersHealthy == 0 via Nomad API monitoring.
When it happens
Trigger: Calling any CSI controller operation (e.g. nomad volume create/delete/detach, snapshot ops) when the controller plugin job has no allocations on ready nodes: all plugin allocations failed, the plugin job is stopped, or every hosting node is down/unresolvable.
Common situations: CSI controller plugin job crashed or was stopped; plugin deployed only on clients that are now down; plugin ID typo/mismatch between volume spec's controller plugin and the job's csi_plugin id; plugin not yet deployed to the cluster.
Related errors
- CSI.ControllerAttachVolume: VolumeID is required
- CSI.ControllerAttachVolume: ClientCSINodeID is required
- CSI.ControllerDetachVolume: VolumeID is required
- CSI.ControllerDetachVolume: ClientCSINodeID is required
- CSI.NodeDetachVolume: PluginID is required
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/f3e65abfa29bb477.
Report an issue: GitHub.