hashicorp/nomad · warning · ErrCSIClientRPCRetryable

CSI.ControllerExpandVolume could not find plugin: %w: %v (wr

Error message

CSI.ControllerExpandVolume could not find plugin: %w: %v (wraps ErrCSIClientRPCRetryable)

What it means

Returned by the CSI ControllerExpandVolume handler when c.findControllerPlugin(req.PluginID) cannot locate a running controller plugin instance on this client. The error wraps nstructs.ErrCSIClientRPCRetryable, signalling the Nomad server that its view of plugin placement/health is stale and the RPC should be retried against another client (or after the plugin registers).

Source

Thrown at client/csi_endpoint.go:243

	// the server RPC call

	resp.Topologies = make([]*nstructs.CSITopology, len(cresp.Volume.AccessibleTopology))
	for _, topo := range cresp.Volume.AccessibleTopology {
		resp.Topologies = append(resp.Topologies,
			&nstructs.CSITopology{Segments: topo.Segments})
	}

	return nil
}

func (c *CSI) ControllerExpandVolume(req *structs.ClientCSIControllerExpandVolumeRequest, resp *structs.ClientCSIControllerExpandVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "expand_volume"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerExpandVolume could not find plugin: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerExpandVolume errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	cresp, err := plugin.ControllerExpandVolume(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the volume was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the volume expansion — the error is explicitly marked retryable and the server should re-route to a healthy controller instance
2. Check the controller plugin job is running on some node (nomad job status <csi-plugin-job>) and restart it if it crashed
3. Verify the pluginID on the volume registration matches the registered CSI plugin (nomad plugin status)
4. Wait for plugin fingerprinting to complete after a node restart before issuing expansions
5. Confirm the controller plugin (not just node plugin) is deployed for the storage backend

Example fix

// before: expanding immediately after plugin upgrade
nomad volume detach/expand ...
// after: verify controller health first, then retry on this error
nomad plugin status <plugin-id>
# if no controllers healthy: nomad job start <csi-controller-job>, then retry expand

// Before expanding, confirm the controller plugin is registered and healthy
// API: GET /v1/volumes/csi/<id> -> check ControllersHealthy > 0
// shell: nomad plugin status <plugin-id>

// The error wraps ErrCSIClientRPCRetryable; detect and retry with backoff
err := client.ExpandVolume(req)
if err != nil && strings.Contains(err.Error(), "could not find plugin") {
    time.Sleep(backoff) // server will re-route to another client instance
    err = client.ExpandVolume(req)
}

When it happens

Trigger: findControllerPlugin fails because the plugin ID in the volume's request is not registered on this node, the controller plugin task has crashed/stopped, or the plugin has not yet finished fingerprinting/registration after client or node startup.

Common situations: CSI controller plugin job stopped or OOM-killed; volume registered with a pluginID typo; node running only the node-plugin (no controller); transient state during cluster startup or plugin upgrade; server routing the expand request to a client that no longer hosts the controller.

Related errors

• CSI.ControllerDeleteVolume: %w: %v
• CSI.ControllerListVolumes: %w: %v
• nil response from plugin.NodeExpandVolume
• expand is not implemented by this controller plugin
• one of LimitBytes or RequiredBytes must be set

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b96d97bdd302a1ae. Report an issue: GitHub.