{"record":{"id":"090912ad77c5a356","repo":"hashicorp/nomad","slug":"controller-list-volumes-v","errorCode":null,"errorMessage":"controller list volumes: %v","messagePattern":"controller list volumes: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":126,"sourceCode":"\t\t\"ClientCSI.ControllerDeleteVolume\",\n\t\tstructs.RateMetricWrite,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller delete volume: %v\", err)\n\t}\n\treturn nil\n}\n\nfunc (a *ClientCSI) ControllerListVolumes(args *cstructs.ClientCSIControllerListVolumesRequest, reply *cstructs.ClientCSIControllerListVolumesResponse) error {\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"client_csi_controller\", \"list_volumes\"}, time.Now())\n\n\terr := a.sendCSIControllerRPC(args.PluginID,\n\t\t\"CSI.ControllerListVolumes\",\n\t\t\"ClientCSI.ControllerListVolumes\",\n\t\tstructs.RateMetricList,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller list volumes: %v\", err)\n\t}\n\treturn nil\n}\n\nfunc (a *ClientCSI) ControllerCreateSnapshot(args *cstructs.ClientCSIControllerCreateSnapshotRequest, reply *cstructs.ClientCSIControllerCreateSnapshotResponse) error {\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"client_csi_controller\", \"create_snapshot\"}, time.Now())\n\n\terr := a.sendCSIControllerRPC(args.PluginID,\n\t\t\"CSI.ControllerCreateSnapshot\",\n\t\t\"ClientCSI.ControllerCreateSnapshot\",\n\t\tstructs.RateMetricWrite,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller create snapshot: %v\", err)\n\t}\n\treturn nil\n}\n","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L108-L144","documentation":"Nomad's server-side ClientCSI.ControllerListVolumes forwards a CSI.ControllerListVolumes RPC to a client node running the controller plugin via sendCSIControllerRPC. When that forwarded RPC fails for any reason (no healthy controller, connection failure, plugin RPC error), the server wraps it with 'controller list volumes: %v'. It indicates the server could not obtain the volume list from the CSI controller plugin.","triggerScenarios":"Calling the ControllerListVolumes RPC (e.g. `nomad volume status` on external volumes) when no healthy controller instance exists for the plugin, the plugin's client connection is down, or the plugin itself returns an RPC error.","commonSituations":"CSI controller plugin job crashed or was stopped; plugin not yet healthy after deployment; network partition between Nomad server and client running the plugin; volume registered with a pluginID that has no controller.","solutions":["Check plugin health: `nomad plugin status <pluginID>` and ensure at least one controller instance is healthy.","Verify the CSI controller plugin job is running: `nomad job status <csi-plugin-job>` and restart it if dead.","Re-register the volume with the correct plugin_id if it points to a nonexistent plugin.","Inspect server/client logs for the underlying sendCSIControllerRPC error to fix the root cause (connection, timeout, plugin error)."],"exampleFix":"// before (volume registered against wrong plugin)\nnomad volume register volume-from-other-cluster.hcl\n// after (re-register with correct plugin_id)\nnomad volume deregister -force ebs-vol\n# edit: plugin_id = \"aws-ebs-controller\" matching the running plugin\nnomad volume register ebs-volume.hcl","handlingStrategy":"retry","validationCode":"// preflight: ensure a healthy controller exists before calling\nstatus, _, err := client.Plugins().Info(ctx, \"aws-ebs-controller\", nil)\nif err != nil || status.ControllersHealthy == 0 {\n    return fmt.Errorf(\"controller plugin not healthy; aborting list volumes\")\n}","typeGuard":"func controllerHealthy(p *api.CSIPlugin) bool {\n    return p != nil && p.ControllersHealthy > 0\n}","tryCatchPattern":"err := csi.ControllerListVolumes(args, reply)\nif err != nil {\n    if strings.Contains(err.Error(), \"controller list volumes:\") {\n        // inspect plugin health and retry with backoff\n    }\n    return err\n}","preventionTips":["Check `nomad plugin status <id>` controller health before volume operations.","Keep the CSI controller job under a restart stanza so crashes self-heal.","Alert on controller alloc crashes / zero healthy controllers.","Pin plugin_id values in volume specs to registered plugins."],"tags":["csi","nomad","rpc-forwarding","storage"],"backgroundTag":"csi-controller-unavailable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}