{"record":{"id":"544763e4c7c082e7","repo":"hashicorp/nomad","slug":"csi-controllerlistvolumes-v","errorCode":null,"errorMessage":"CSI.ControllerListVolumes: %v","messagePattern":"CSI\\.ControllerListVolumes: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":336,"sourceCode":"\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerListVolumes: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq := req.ToCSIRequest()\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerListVolumes errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\tcresp, err := plugin.ControllerListVolumes(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerListVolumes: %v\", err)\n\t}\n\n\tresp.NextToken = cresp.NextToken\n\tresp.Entries = []*nstructs.CSIVolumeExternalStub{}\n\n\tfor _, entry := range cresp.Entries {\n\t\tif entry.Volume == nil {\n\t\t\treturn fmt.Errorf(\"CSI.ControllerListVolumes: plugin returned an invalid entry\")\n\t\t}\n\t\tvol := &nstructs.CSIVolumeExternalStub{\n\t\t\tExternalID:    entry.Volume.ExternalVolumeID,\n\t\t\tCapacityBytes: entry.Volume.CapacityBytes,\n\t\t\tVolumeContext: entry.Volume.VolumeContext,\n\t\t\tCloneID:       entry.Volume.ContentSource.CloneID,\n\t\t\tSnapshotID:    entry.Volume.ContentSource.SnapshotID,\n\t\t}\n\t\tif entry.Status != nil {\n\t\t\tvol.PublishedExternalNodeIDs = entry.Status.PublishedNodeIds","sourceCodeStart":318,"sourceCodeEnd":354,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L318-L354","documentation":"The CSI plugin's ControllerListVolumes RPC failed after Nomad's retry wrapper (3 attempts, exponential backoff, per-retry timeout CSIPluginRequestTimeout). Only ResourceExhausted-type codes are retried; any other error is returned wrapped as 'CSI.ControllerListVolumes: <err>' and treated as fatal for that RPC.","triggerScenarios":"The plugin is unreachable (socket down/crashed), returns gRPC codes like Unavailable, Unimplemented, PermissionDenied, or InvalidArgument; per-attempt timeout exceeded on a slow backend; the cresp nil-handling is not involved here — any non-nil err is wrapped.","commonSituations":"Enumerating external volumes ('nomad volume status' external view) while the controller is overloaded or restarting; plugin lacks permissions to list volumes in the cloud account; plugin does not implement ListVolumes (Unimplemented); storage backend throttling causes timeouts.","solutions":["Verify the controller plugin container/task is running and its socket is reachable from the Nomad client","Check the wrapped gRPC code in plugin logs (e.g. Unimplemented means upgrade the plugin; PermissionDenied means fix cloud credentials)","Increase CSIPluginRequestTimeout tolerance or scale the backend if ResourceExhausted/throttling persists","Re-run the listing after the plugin stabilizes"],"exampleFix":"// before: controller plugin job crashed, list fails\njob \"ebs-controller\" { ... count = 0 }\n// after\njob \"ebs-controller\" {\n  group \"controller\" {\n    task \"plugin\" { ... }\n  }\n}","handlingStrategy":"retry","validationCode":"plug, err := apiClient.Plugins().Get(\"aws-ebs-controller\")\nif err != nil || plug.ControllersHealthy < 1 {\n    return fmt.Errorf(\"controller not available for ListVolumes\")\n}\nif !plug.ControllerInfo.HasCapability(structs.CSIControllerSupportsListVolumes) {\n    return fmt.Errorf(\"plugin does not implement ListVolumes\")\n}","typeGuard":"func isListVolumesFatal(err error) bool {\n    if err == nil { return false }\n    var st *status.Status\n    if errors.As(err, &st) {\n        return st.Code() != codes.ResourceExhausted\n    }\n    return true\n}","tryCatchPattern":"err := client.CSI().ControllerListVolumes(req)\nif err != nil {\n    var st *status.Status\n    if errors.As(err, &st) && st.Code() == codes.ResourceExhausted {\n        // backend throttled: back off and retry\n        time.Sleep(5 * time.Second)\n        return retryListVolumes(req)\n    }\n    if errors.As(err, &st) && st.Code() == codes.Unimplemented {\n        return fmt.Errorf(\"plugin lacks ListVolumes support: %w\", err)\n    }\n    return err\n}","preventionTips":["Verify the plugin advertises ListVolumes capability before relying on external volume listing","Keep plugin cloud credentials valid to avoid PermissionDenied","Size the controller/backend to avoid ResourceExhausted throttling; add backoff on that code","Monitor plugin task health and restarts; a crashing plugin is the most common root cause"],"tags":["csi","nomad","grpc","list-volumes"],"backgroundTag":"csi-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}