hashicorp/nomad · error · ErrCSIClientRPCRetryable

CSI.ControllerAttachVolume: %w: %v (wraps ErrCSIClientRPCRet

Error message

CSI.ControllerAttachVolume: %w: %v (wraps ErrCSIClientRPCRetryable)

What it means

ControllerAttachVolume wraps a findControllerPlugin failure in ErrCSIClientRPCRetryable, signaling the server that its view of plugin health is stale and it should retry with another controller instance. The wrapped cause is why no controller plugin could be obtained on this client.

Source

Thrown at client/csi_endpoint.go:96

	}
	return nil
}

// ControllerAttachVolume is used to attach a volume from a CSI Cluster to
// the storage node provided in the request.
//
// The controller attachment flow currently works as follows:
// 1. Validate the volume request
// 2. Call ControllerPublishVolume on the CSI Plugin to trigger a remote attachment
//
// In the future this may be expanded to request dynamic secrets for attachment.
func (c *CSI) ControllerAttachVolume(req *structs.ClientCSIControllerAttachVolumeRequest, resp *structs.ClientCSIControllerAttachVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "publish_volume"}, time.Now())
	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerAttachVolume: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	// The following block of validation checks should not be reached on a
	// real Nomad cluster as all of this data should be validated when registering
	// volumes with the cluster. They serve as a defensive check before forwarding
	// requests to plugins, and to aid with development.

	if req.VolumeID == "" {
		return errors.New("CSI.ControllerAttachVolume: VolumeID is required")
	}

	if req.ClientCSINodeID == "" {
		return errors.New("CSI.ControllerAttachVolume: ClientCSINodeID is required")
	}

	csiReq, err := req.ToCSIRequest()

View on GitHub (pinned to 482b49bf1a)

Solutions

Check `nomad plugin status` and allocation state for the controller plugin job; restart/reschedule it if unhealthy
Retry the publish/unpublish workflow — the sentinel makes it retryable at the server
Ensure the plugin is registered as a controller (not node-only) and req.PluginID matches
Update volume registration to point at the correct plugin ID if the plugin was renamed

Example fix

// before
plugin_id = "efs-plugin" // job now registers as "efs-csi-controller"
// after
plugin_id = "efs-csi-controller" # match job's plugin stanza

Defensive patterns

Strategy: retry

Validate before calling

// before publish, confirm the controller plugin is healthy on this client
if p := clientCSIPlugin(pluginID); p == nil || p.Healthy() != nil {
    return structs.NewErrRPCCallFailed(clientAddr, "controller plugin unavailable")
}

Type guard

func isRetryableCSI(err error) bool {
    return structs.IsErrRetryable(err)
}

Try / catch

err := c.ControllerAttachVolume(req, resp)
if err != nil {
    if structs.IsErrRetryable(err) {
        // retry against another controller instance / schedule another node
        return structs.NewErrRPCCallFailed(addr, err.Error())
    }
    return err
}

Prevention

Run controller plugins on dedicated, monitored nodes
Set restart/service-check policies so crashed plugins recover automatically
Pin plugin_id consistently in job volume blocks and volume registrations
Before publishing, detach stale attachments at the provider

When it happens

Trigger: A ClientCSIControllerAttachVolume (publish) RPC targets a client where req.PluginID does not resolve to a healthy controller plugin in the client's plugin registry.

Common situations: Controller plugin job stopped, crashed, or rescheduled elsewhere; stale server plugin-health cache; plugin registered as node-only (no controller) so ControllerPublishVolume is unsupported; plugin ID typo after renaming the plugin job.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9ded049ff894b10f. Report an issue: GitHub.