{"record":{"id":"e1ec3a8c7ba3bc01","repo":"hashicorp/nomad","slug":"csi-controllervalidatevolume-w-v-wraps-errcsi","errorCode":null,"errorMessage":"CSI.ControllerValidateVolume: %w: %v (wraps ErrCSIClientRPCRetryable)","messagePattern":"CSI\\.ControllerValidateVolume: %w: (.+?) \\(wraps ErrCSIClientRPCRetryable\\)","errorType":"exception","errorClass":"ErrCSIClientRPCRetryable","httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":57,"sourceCode":"// ControllerValidateVolume is used during volume registration to validate\n// that a volume exists and that the capabilities it was registered with are\n// supported by the CSI Plugin and external volume configuration.\nfunc (c *CSI) ControllerValidateVolume(req *structs.ClientCSIControllerValidateVolumeRequest, resp *structs.ClientCSIControllerValidateVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"validate_volume\"}, time.Now())\n\n\tif req.VolumeID == \"\" {\n\t\treturn errors.New(\"CSI.ControllerValidateVolume: VolumeID is required\")\n\t}\n\n\tif req.PluginID == \"\" {\n\t\treturn errors.New(\"CSI.ControllerValidateVolume: PluginID is required\")\n\t}\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerValidateVolume: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq, err := req.ToCSIRequest()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerValidateVolume: %v\", err)\n\t}\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ValidateVolumeCapabilities errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\terr = plugin.ControllerValidateCapabilities(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L39-L75","documentation":"ControllerValidateVolume wraps a findControllerPlugin failure in ErrCSIClientRPCRetryable so the Nomad server learns the plugin-health view is stale and can retry with another controller instance. The %w wraps the retryable sentinel; %v carries the underlying cause (plugin not found / not controller-capable / not healthy on this client).","triggerScenarios":"A ClientCSIControllerValidateVolume RPC arrives at a client whose CSI plugin registry has no healthy controller plugin matching req.PluginID, so findControllerPlugin fails before any CSI RPC is made.","commonSituations":"Plugin was deregistered/restarted (job updated or plugin task crashed) while server still routes to this client; server-side plugin health cache is stale; plugin ID typo or controller-only vs node-only plugin mismatch after a plugin version change.","solutions":["Verify the plugin job is running and healthy on the intended client (`nomad plugin status`, check allocations)","Re-register the volume/ensure the server refreshes plugin health, then retry the operation — the error is intentionally retryable","Confirm req.PluginID matches the plugin's registered ID (plugin type must be controller-capable)","If the plugin moved to another node, let Nomad reschedule/route to the healthy controller instance"],"exampleFix":"// before\nif err := vol.Validate(ctx); err != nil { return err } // treating it as fatal\n// after\nif err := vol.Validate(ctx); err != nil {\n    if structs.IsErrRetryable(err) { /* server will retry with another controller */ }\n    return err\n}","handlingStrategy":"retry","validationCode":"// before calling validate, check plugin health server-side\nplugin := server.GetPluginByClient(clientID, pluginID)\nif plugin == nil || !plugin.ControllerRequired() || plugin.Healthy() != nil {\n    return fmt.Errorf(\"controller plugin %s not healthy on client %s\", pluginID, clientID)\n}","typeGuard":"func isRetryableCSI(err error) bool {\n    var rErr *structs.RetryableError\n    return errors.As(err, &rErr)\n}","tryCatchPattern":"err := c.ControllerValidateVolume(req, resp)\nif err != nil {\n    if structs.IsErrRetryable(err) {\n        // reschedule to another controller instance / retry with backoff\n        return structs.NewErrRPCCallFailed(agentAddr, err.Error())\n    }\n    return err // fatal: inspect plugin ID and volume registration\n}","preventionTips":["Keep controller plugin jobs always-on (restart stanza, service checks)","Check `nomad plugin status` before volume workflows","Keep plugin IDs stable across job updates","Reconcile plugin health regularly so the server cache stays fresh"],"tags":["csi","plugin","retryable","controller"],"backgroundTag":"csi-plugin-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}