{"record":{"id":"d71ca80d8e2f0dde","repo":"hashicorp/nomad","slug":"csi-controllerlistvolumes-w-v","errorCode":null,"errorMessage":"CSI.ControllerListVolumes: %w: %v","messagePattern":"CSI\\.ControllerListVolumes: %w: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":319,"sourceCode":"\t\t// if the volume was deleted out-of-band, we'll get an error from\n\t\t// the plugin but can safely ignore it\n\t\tc.c.logger.Debug(\"could not delete volume\", \"error\", err)\n\t\treturn nil\n\t}\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerDeleteVolume: %v\", err)\n\t}\n\treturn err\n}\n\nfunc (c *CSI) ControllerListVolumes(req *structs.ClientCSIControllerListVolumesRequest, resp *structs.ClientCSIControllerListVolumesResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"list_volumes\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerListVolumes: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq := req.ToCSIRequest()\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerListVolumes errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\tcresp, err := plugin.ControllerListVolumes(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerListVolumes: %v\", err)\n\t}","sourceCodeStart":301,"sourceCodeEnd":337,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L301-L337","documentation":"Nomad's ControllerListVolumes endpoint failed at findControllerPlugin: no healthy controller plugin instance matching req.PluginID was found on the client. Like the other find failures it is wrapped with structs.ErrCSIClientRPCRetryable so the server treats it as a stale-plugin-health condition and retries, potentially routing to another controller.","triggerScenarios":"Running 'nomad volume status' / list-volumes external enumeration while the controller plugin is down, restarting, failed health checks, or the plugin ID doesn't match a registered controller; the request may also target a client node that doesn't host the controller.","commonSituations":"Listing external volumes during a plugin upgrade; controller job crashed; controller only scheduled on some nodes but the request hit one without it; stale plugin ID after renaming the plugin job.","solutions":["Retry — the RPC is marked retryable and will succeed once a healthy controller registers","Check the controller plugin job health (nomad job status <plugin>; nomad plugin status <plugin-id>)","Ensure the plugin_id used in the query/volume spec matches the registered controller plugin","Restart or reschedule the controller plugin job if it is stuck unhealthy"],"exampleFix":"// before: querying with wrong plugin id\nnomad plugin status csi-ebs-demo-controller-typo\n// after\nnomad plugin status aws-ebs-controller","handlingStrategy":"retry","validationCode":"plug, err := apiClient.Plugins().Get(\"aws-ebs-controller\")\nif err != nil || plug.ControllersHealthy < 1 {\n    return fmt.Errorf(\"controller unavailable; cannot list volumes\")\n}","typeGuard":"func isCSIRetryable(err error) bool {\n    return err != nil && errors.Is(err, structs.ErrCSIClientRPCRetryable)\n}","tryCatchPattern":"err := client.CSI().ControllerListVolumes(req)\nif errors.Is(err, structs.ErrCSIClientRPCRetryable) {\n    time.Sleep(2 * time.Second)\n    return retryListVolumes(req)\n}\nreturn err","preventionTips":["Ensure the controller plugin job is scheduled and healthy before querying","Use the correct plugin_id (verify with nomad plugin status)","Deploy controllers on enough nodes / with proper constraints so one is always reachable","Avoid plugin upgrades that drop controller count to zero during operations"],"tags":["csi","nomad","plugin-not-found","retryable"],"backgroundTag":"csi-plugin-unavailable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}