hashicorp/nomad · error

CSI.ControllerListVolumes: %w: %v

Error message

CSI.ControllerListVolumes: %w: %v

What it means

Nomad's ControllerListVolumes endpoint failed at findControllerPlugin: no healthy controller plugin instance matching req.PluginID was found on the client. Like the other find failures it is wrapped with structs.ErrCSIClientRPCRetryable so the server treats it as a stale-plugin-health condition and retries, potentially routing to another controller.

Source

Thrown at client/csi_endpoint.go:319

		// if the volume was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it
		c.c.logger.Debug("could not delete volume", "error", err)
		return nil
	}
	if err != nil {
		return fmt.Errorf("CSI.ControllerDeleteVolume: %v", err)
	}
	return err
}

func (c *CSI) ControllerListVolumes(req *structs.ClientCSIControllerListVolumesRequest, resp *structs.ClientCSIControllerListVolumesResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "list_volumes"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerListVolumes: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerListVolumes errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	cresp, err := plugin.ControllerListVolumes(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if err != nil {
		return fmt.Errorf("CSI.ControllerListVolumes: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry — the RPC is marked retryable and will succeed once a healthy controller registers
2. Check the controller plugin job health (nomad job status <plugin>; nomad plugin status <plugin-id>)
3. Ensure the plugin_id used in the query/volume spec matches the registered controller plugin
4. Restart or reschedule the controller plugin job if it is stuck unhealthy

Example fix

// before: querying with wrong plugin id
nomad plugin status csi-ebs-demo-controller-typo
// after
nomad plugin status aws-ebs-controller

plug, err := apiClient.Plugins().Get("aws-ebs-controller")
if err != nil || plug.ControllersHealthy < 1 {
    return fmt.Errorf("controller unavailable; cannot list volumes")
}

func isCSIRetryable(err error) bool {
    return err != nil && errors.Is(err, structs.ErrCSIClientRPCRetryable)
}

err := client.CSI().ControllerListVolumes(req)
if errors.Is(err, structs.ErrCSIClientRPCRetryable) {
    time.Sleep(2 * time.Second)
    return retryListVolumes(req)
}
return err

When it happens

Trigger: Running 'nomad volume status' / list-volumes external enumeration while the controller plugin is down, restarting, failed health checks, or the plugin ID doesn't match a registered controller; the request may also target a client node that doesn't host the controller.

Common situations: Listing external volumes during a plugin upgrade; controller job crashed; controller only scheduled on some nodes but the request hit one without it; stale plugin ID after renaming the plugin job.

Related errors

• CSI.ControllerDeleteVolume: %w: %v
• CSI.ControllerExpandVolume could not find plugin: %w: %v (wr
• %v: %v
• no such plugin
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d71ca80d8e2f0dde. Report an issue: GitHub.