hashicorp/nomad · error
no healthy controllers for CSI plugin: %s
Error message
no healthy controllers for CSI plugin: %s
What it means
Fires in pluginValidateVolume: the CSI plugin exists and requires a controller, but ControllersHealthy is zero — no controller instances of the plugin are currently healthy, so volume operations needing a controller cannot proceed.
Source
Thrown at nomad/csi_endpoint.go:230
reply.Volume = vol
return v.srv.replySetIndex(csiVolumeTable, &reply.QueryMeta)
}}
return v.srv.blockingRPC(&opts)
}
func (v *CSIVolume) pluginValidateVolume(vol *structs.CSIVolume) (*structs.CSIPlugin, error) {
state := v.srv.fsm.State()
plugin, err := state.CSIPluginByID(nil, vol.PluginID)
if err != nil {
return nil, err
}
if plugin == nil {
return nil, fmt.Errorf("no CSI plugin named: %s could be found", vol.PluginID)
}
if plugin.ControllerRequired && plugin.ControllersHealthy < 1 {
return nil, fmt.Errorf("no healthy controllers for CSI plugin: %s", vol.PluginID)
}
vol.Provider = plugin.Provider
vol.ProviderVersion = plugin.Version
return plugin, nil
}
func (v *CSIVolume) controllerValidateVolume(req *structs.CSIVolumeRegisterRequest, vol *structs.CSIVolume, plugin *structs.CSIPlugin) error {
if !plugin.ControllerRequired {
// The plugin does not require a controller, so for now we won't do any
// further validation of the volume.
return nil
}
method := "ClientCSI.ControllerValidateVolume"
cReq := &cstructs.ClientCSIControllerValidateVolumeRequest{View on GitHub (pinned to 482b49bf1a)
Solutions
- Check nomad plugin status <plugin> for controller health and restart the CSI controller job
- Fix scheduling constraints so the controller can be placed on an eligible node
- Wait for plugin health to update after (re)starting the controller, then retry
Defensive patterns
Strategy: validation
Validate before calling
p, _ := client.CSIPlugins().Get(pluginID, nil)
if p != nil && p.ControllersHealthy < 1 {
return errors.New("controller unhealthy; abort volume registration")
} Type guard
func controllersHealthy(p *api.CSIPlugin) bool { return p != nil && p.ControllersHealthy >= 1 } Try / catch
if err != nil && strings.Contains(err.Error(), "no healthy controllers") { wait/restart controller then retry } Prevention
- Monitor plugin controller health before volume operations
- Set correct datacenter/constraints so controllers schedule
- Retry after controller (re)registration with backoff
When it happens
Trigger: Registering or creating a volume on a plugin with ControllerRequired=true when no controller healthy count is >= 1, typically because controller instances are down or just registered.
Common situations: CSI controller task crashed or OOMed; controller not scheduled due to node constraints; plugin just started and health hasn't been established yet; cluster after node failure.
Related errors
- CSI Plugin loaded incorrectly
- CSI.ControllerValidateVolume: PluginID is required
- could not validate task driver capabilities: %v
- CSI plugin failed probe: %w
- CSI.ControllerValidateVolume: %w: %v (wraps ErrCSIClientRPCR
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/d36cb73d47cc8893.
Report an issue: GitHub.