hashicorp/nomad · error · ErrCSIClientRPCRetryable

CSI.ControllerValidateVolume: %w: %v (wraps ErrCSIClientRPCR

Error message

CSI.ControllerValidateVolume: %w: %v (wraps ErrCSIClientRPCRetryable)

What it means

ControllerValidateVolume wraps a findControllerPlugin failure in ErrCSIClientRPCRetryable so the Nomad server learns the plugin-health view is stale and can retry with another controller instance. The %w wraps the retryable sentinel; %v carries the underlying cause (plugin not found / not controller-capable / not healthy on this client).

Source

Thrown at client/csi_endpoint.go:57

// ControllerValidateVolume is used during volume registration to validate
// that a volume exists and that the capabilities it was registered with are
// supported by the CSI Plugin and external volume configuration.
func (c *CSI) ControllerValidateVolume(req *structs.ClientCSIControllerValidateVolumeRequest, resp *structs.ClientCSIControllerValidateVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "validate_volume"}, time.Now())

	if req.VolumeID == "" {
		return errors.New("CSI.ControllerValidateVolume: VolumeID is required")
	}

	if req.PluginID == "" {
		return errors.New("CSI.ControllerValidateVolume: PluginID is required")
	}

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerValidateVolume: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq, err := req.ToCSIRequest()
	if err != nil {
		return fmt.Errorf("CSI.ControllerValidateVolume: %v", err)
	}

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ValidateVolumeCapabilities errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	err = plugin.ControllerValidateCapabilities(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the plugin job is running and healthy on the intended client (`nomad plugin status`, check allocations)
2. Re-register the volume/ensure the server refreshes plugin health, then retry the operation — the error is intentionally retryable
3. Confirm req.PluginID matches the plugin's registered ID (plugin type must be controller-capable)
4. If the plugin moved to another node, let Nomad reschedule/route to the healthy controller instance

Example fix

// before
if err := vol.Validate(ctx); err != nil { return err } // treating it as fatal
// after
if err := vol.Validate(ctx); err != nil {
    if structs.IsErrRetryable(err) { /* server will retry with another controller */ }
    return err
}

// before calling validate, check plugin health server-side
plugin := server.GetPluginByClient(clientID, pluginID)
if plugin == nil || !plugin.ControllerRequired() || plugin.Healthy() != nil {
    return fmt.Errorf("controller plugin %s not healthy on client %s", pluginID, clientID)
}

func isRetryableCSI(err error) bool {
    var rErr *structs.RetryableError
    return errors.As(err, &rErr)
}

err := c.ControllerValidateVolume(req, resp)
if err != nil {
    if structs.IsErrRetryable(err) {
        // reschedule to another controller instance / retry with backoff
        return structs.NewErrRPCCallFailed(agentAddr, err.Error())
    }
    return err // fatal: inspect plugin ID and volume registration
}

When it happens

Trigger: A ClientCSIControllerValidateVolume RPC arrives at a client whose CSI plugin registry has no healthy controller plugin matching req.PluginID, so findControllerPlugin fails before any CSI RPC is made.

Common situations: Plugin was deregistered/restarted (job updated or plugin task crashed) while server still routes to this client; server-side plugin health cache is stale; plugin ID typo or controller-only vs node-only plugin mismatch after a plugin version change.

Related errors

• CSI.ControllerAttachVolume: %w: %v (wraps ErrCSIClientRPCRet
• CSI.ControllerDetachVolume: %w: %v (wraps ErrCSIClientRPCRet
• CSI.ControllerCreateVolume: %w: %v (wraps ErrCSIClientRPCRet
• cannot upload management tokens
• CSI Plugin loaded incorrectly

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e1ec3a8c7ba3bc01. Report an issue: GitHub.