{"record":{"id":"13ac3717427ee109","repo":"hashicorp/nomad","slug":"csi-controllercreatevolume-w-v-wraps-errcsicl","errorCode":null,"errorMessage":"CSI.ControllerCreateVolume: %w: %v (wraps ErrCSIClientRPCRetryable)","messagePattern":"CSI\\.ControllerCreateVolume: %w: (.+?) \\(wraps ErrCSIClientRPCRetryable\\)","errorType":"exception","errorClass":"ErrCSIClientRPCRetryable","httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":192,"sourceCode":"\t\t// if the controller detach previously happened but the server failed to\n\t\t// checkpoint, we'll get an error from the plugin but can safely ignore it.\n\t\tc.c.logger.Debug(\"could not unpublish volume\", \"error\", err)\n\t\treturn nil\n\t}\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerDetachVolume: %v\", err)\n\t}\n\treturn err\n}\n\nfunc (c *CSI) ControllerCreateVolume(req *structs.ClientCSIControllerCreateVolumeRequest, resp *structs.ClientCSIControllerCreateVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"create_volume\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerCreateVolume: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq, err := req.ToCSIRequest()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerCreateVolume: %v\", err)\n\t}\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerCreateVolume errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\tcresp, err := plugin.ControllerCreateVolume(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L174-L210","documentation":"ControllerCreateVolume wraps a findControllerPlugin failure in ErrCSIClientRPCRetryable, telling the server its plugin-health view is stale and it should retry with another controller instance. The underlying err explains why no controller plugin matched req.PluginID.","triggerScenarios":"A ClientCSIControllerCreateVolume RPC (from `nomad volume create` or external volume registration) reaches a client where no healthy controller plugin exists for req.PluginID.","commonSituations":"Dynamic volume provisioning while the controller plugin job is starting/crashed; plugin registered node-only so it cannot create volumes; cluster scale-down removed the node hosting the controller; plugin ID mismatch after job rename.","solutions":["Ensure the controller plugin job is running and healthy (`nomad plugin status <plugin>`), then retry volume creation — the error is deliberately retryable","Check the plugin is controller-capable (type controller) and the plugin ID in the volume spec matches","Restart or reschedule the plugin allocation if it is unhealthy","If the controller moved, allow the server to re-route to the healthy controller instance"],"exampleFix":"// before\nvolume {\n  type = \"csi\"\n  plugin_id = \"ebs\"   # job registers plugin as \"ebs-csi\"\n}\n// after\nvolume {\n  type = \"csi\"\n  plugin_id = \"ebs-csi\"\n}","handlingStrategy":"retry","validationCode":"// before volume create, confirm a healthy controller plugin is registered\np := serverCSIPlugin(pluginID)\nif p == nil || p.ControllerRequired() == false || p.Healthy() != nil {\n    return fmt.Errorf(\"controller plugin %s not available for volume creation\", pluginID)\n}","typeGuard":"func isRetryableCSI(err error) bool { return structs.IsErrRetryable(err) }","tryCatchPattern":"err := c.ControllerCreateVolume(req, resp)\nif err != nil {\n    if structs.IsErrRetryable(err) {\n        // wait for plugin health to refresh, then retry with backoff\n        time.Sleep(retryDelay)\n        return c.ControllerCreateVolume(req, resp)\n    }\n    return err\n}","preventionTips":["Start controller plugin jobs before running `nomad volume create`","Verify plugin_id in the volume spec matches the plugin job's plugin stanza","Monitor plugin allocations for restarts during provisioning","Retry create operations with backoff since the sentinel is intentionally retryable"],"tags":["csi","plugin","retryable","create-volume"],"backgroundTag":"csi-plugin-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}