What causes "controller list volumes: %v" in hashicorp/nomad?

Check plugin health: `nomad plugin status ` and ensure at least one controller instance is healthy.

controller list volumes: %v

Error message

controller list volumes: %v

What it means

Nomad's server-side ClientCSI.ControllerListVolumes forwards a CSI.ControllerListVolumes RPC to a client node running the controller plugin via sendCSIControllerRPC. When that forwarded RPC fails for any reason (no healthy controller, connection failure, plugin RPC error), the server wraps it with 'controller list volumes: %v'. It indicates the server could not obtain the volume list from the CSI controller plugin.

Source

Thrown at nomad/client_csi_endpoint.go:126

		"ClientCSI.ControllerDeleteVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller delete volume: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerListVolumes(args *cstructs.ClientCSIControllerListVolumesRequest, reply *cstructs.ClientCSIControllerListVolumesResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "list_volumes"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerListVolumes",
		"ClientCSI.ControllerListVolumes",
		structs.RateMetricList,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller list volumes: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerCreateSnapshot(args *cstructs.ClientCSIControllerCreateSnapshotRequest, reply *cstructs.ClientCSIControllerCreateSnapshotResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "create_snapshot"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerCreateSnapshot",
		"ClientCSI.ControllerCreateSnapshot",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller create snapshot: %v", err)
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Check plugin health: `nomad plugin status <pluginID>` and ensure at least one controller instance is healthy.
Verify the CSI controller plugin job is running: `nomad job status <csi-plugin-job>` and restart it if dead.
Re-register the volume with the correct plugin_id if it points to a nonexistent plugin.
Inspect server/client logs for the underlying sendCSIControllerRPC error to fix the root cause (connection, timeout, plugin error).

Example fix

// before (volume registered against wrong plugin)
nomad volume register volume-from-other-cluster.hcl
// after (re-register with correct plugin_id)
nomad volume deregister -force ebs-vol
# edit: plugin_id = "aws-ebs-controller" matching the running plugin
nomad volume register ebs-volume.hcl

Defensive patterns

Strategy: retry

Validate before calling

// preflight: ensure a healthy controller exists before calling
status, _, err := client.Plugins().Info(ctx, "aws-ebs-controller", nil)
if err != nil || status.ControllersHealthy == 0 {
    return fmt.Errorf("controller plugin not healthy; aborting list volumes")
}

Type guard

func controllerHealthy(p *api.CSIPlugin) bool {
    return p != nil && p.ControllersHealthy > 0
}

Try / catch

err := csi.ControllerListVolumes(args, reply)
if err != nil {
    if strings.Contains(err.Error(), "controller list volumes:") {
        // inspect plugin health and retry with backoff
    }
    return err
}

Prevention

Check `nomad plugin status <id>` controller health before volume operations.
Keep the CSI controller job under a restart stanza so crashes self-heal.
Alert on controller alloc crashes / zero healthy controllers.
Pin plugin_id values in volume specs to registered plugins.

When it happens

Trigger: Calling the ControllerListVolumes RPC (e.g. `nomad volume status` on external volumes) when no healthy controller instance exists for the plugin, the plugin's client connection is down, or the plugin itself returns an RPC error.

Common situations: CSI controller plugin job crashed or was stopped; plugin not yet healthy after deployment; network partition between Nomad server and client running the plugin; volume registered with a pluginID that has no controller.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/090912ad77c5a356. Report an issue: GitHub.