hashicorp/nomad · error · ErrCSIClientRPCRetryable

CSI.ControllerDetachVolume: %w: %v (wraps ErrCSIClientRPCRet

Error message

CSI.ControllerDetachVolume: %w: %v (wraps ErrCSIClientRPCRetryable)

What it means

ControllerDetachVolume wraps a findControllerPlugin failure in ErrCSIClientRPCRetryable so the server knows the plugin-health view is stale and can retry with another controller instance. The cause is the absence of a healthy controller plugin for req.PluginID on this client.

Source

Thrown at client/csi_endpoint.go:144

		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if err != nil {
		return fmt.Errorf("CSI.ControllerAttachVolume: %v", err)
	}

	resp.PublishContext = cresp.PublishContext
	return nil
}

// ControllerDetachVolume is used to detach a volume from a CSI Cluster from
// the storage node provided in the request.
func (c *CSI) ControllerDetachVolume(req *structs.ClientCSIControllerDetachVolumeRequest, resp *structs.ClientCSIControllerDetachVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "unpublish_volume"}, time.Now())
	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerDetachVolume: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	// The following block of validation checks should not be reached on a
	// real Nomad cluster as all of this data should be validated when registering
	// volumes with the cluster. They serve as a defensive check before forwarding
	// requests to plugins, and to aid with development.

	if req.VolumeID == "" {
		return errors.New("CSI.ControllerDetachVolume: VolumeID is required")
	}

	if req.ClientCSINodeID == "" {
		return errors.New("CSI.ControllerDetachVolume: ClientCSINodeID is required")
	}

	csiReq := req.ToCSIRequest()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restore/restart the controller plugin job on the cluster so findControllerPlugin can locate it
2. Retry the detach — the retryable sentinel tells the server to attempt another controller instance
3. If the attachment is orphaned, use the provider CLI or `nomad volume detach` with the correct node/plugin to force detach
4. Verify plugin ID consistency between volume registration and the running plugin job

Example fix

// before
nomad job stop efs-controller   # detach now fails with retryable plugin error
// after
nomad job start efs-controller && nomad volume detach <vol> <node>

// before detach, verify controller plugin availability
if p := clientCSIPlugin(pluginID); p == nil {
    return structs.NewErrRPCCallFailed(clientAddr, "controller plugin gone; retry elsewhere")
}

func isRetryableCSI(err error) bool { return structs.IsErrRetryable(err) }

err := c.ControllerDetachVolume(req, resp)
if err != nil {
    if structs.IsErrRetryable(err) {
        // re-route to another healthy controller
        return structs.NewErrRPCCallFailed(addr, err.Error())
    }
    return err
}

When it happens

Trigger: An unpublish/detach RPC arrives while the controller plugin is deregistered, unhealthy, or scheduled on a different client than the one handling req.PluginID.

Common situations: Controller plugin crashed between attach and detach (leaving a stale attachment); plugin job stopped during maintenance; server retrying unpublish against a node where the plugin no longer runs.

Related errors

• CSI.ControllerValidateVolume: %w: %v (wraps ErrCSIClientRPCR
• CSI.ControllerAttachVolume: %w: %v (wraps ErrCSIClientRPCRet
• CSI.ControllerCreateVolume: %w: %v (wraps ErrCSIClientRPCRet
• controller detach volume: %v
• could not detach from node: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/39d0f57dd763999c. Report an issue: GitHub.