{"record":{"id":"4213eca757c16088","repo":"hashicorp/nomad","slug":"csi-controllerdeletevolume-v","errorCode":null,"errorMessage":"CSI.ControllerDeleteVolume: %v","messagePattern":"CSI\\.ControllerDeleteVolume: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":307,"sourceCode":"\tcsiReq := req.ToCSIRequest()\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerDeleteVolume errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\terr = plugin.ControllerDeleteVolume(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {\n\t\t// if the volume was deleted out-of-band, we'll get an error from\n\t\t// the plugin but can safely ignore it\n\t\tc.c.logger.Debug(\"could not delete volume\", \"error\", err)\n\t\treturn nil\n\t}\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerDeleteVolume: %v\", err)\n\t}\n\treturn err\n}\n\nfunc (c *CSI) ControllerListVolumes(req *structs.ClientCSIControllerListVolumesRequest, resp *structs.ClientCSIControllerListVolumesResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"list_volumes\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerListVolumes: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq := req.ToCSIRequest()\n","sourceCodeStart":289,"sourceCodeEnd":325,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L289-L325","documentation":"The CSI plugin's ControllerDeleteVolume RPC returned an error that is neither a NOT_FOUND-style code (which would be logged as an out-of-band deletion and ignored) nor retried out; Nomad surfaces it wrapped as 'CSI.ControllerDeleteVolume: <err>'. It means the plugin itself refused or failed the delete operation.","triggerScenarios":"The plugin returns codes other than NotFound/ignorable ones — e.g. Internal errors from the storage backend, FailedPrecondition when the volume is in use, InvalidArgument for a bad external volume ID, or transient gRPC failures after the 3 retries with backoff are exhausted.","commonSituations":"Volume still attached/mounted by nodes when delete is issued; underlying storage backend (EBS, Ceph, etc.) rejects deletion; volume was created outside Nomad so the external ID does not exist in the backend; permission/credential problems for the plugin's cloud account.","solutions":["Read the wrapped plugin error and the plugin logs to identify the underlying storage-backend failure","Ensure the volume is not still attached/mounted (check nomad volume status, detach allocations or wait for node unpublish)","Verify the volume's external/source ID exists in the storage backend; if it was already deleted out-of-band, the volume can safely be deregistered from Nomad","Fix backend-side issues (credentials, quotas, in-use snapshots) and retry the delete"],"exampleFix":"// before: deleting while volume still in use\nnomad volume delete -volume-id mysql-data\n// after: stop allocations using it first\nnomad job stop mysql\nnomad volume detach -node-id <node> -volume-id mysql-data\nnomad volume delete -volume-id mysql-data","handlingStrategy":"try-catch","validationCode":"vols, _, err := apiClient.Volumes().List(nil)\nif err == nil {\n    for _, v := range vols {\n        if v.ID == volID && v.Schedulable && v.ControllersHealthy > 0 {\n            // check no live allocations claim it\n            detail, _, err := apiClient.Volumes().Get(volID, nil)\n            if err == nil && len(detail.Allocations) > 0 {\n                return fmt.Errorf(\"volume %s still has allocations; stop jobs before delete\", volID)\n            }\n        }\n    }\n}","typeGuard":"func isVolumeInUseErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"FailedPrecondition\")\n}","tryCatchPattern":"err := client.CSI().ControllerDeleteVolume(req)\nif err != nil {\n    var st *status.Status\n    if errors.As(err, &st) && st.Code() == codes.NotFound {\n        // volume already deleted out-of-band: safe to ignore/deregister\n        return nil\n    }\n    if isVolumeInUseErr(err) {\n        return fmt.Errorf(\"detach volume from all nodes before delete: %w\", err)\n    }\n    return err\n}","preventionTips":["Stop or migrate all allocations using the volume before deleting","Confirm the volume's ExternalID exists in the storage backend before delete","Keep plugin cloud credentials and quotas valid","Check plugin logs for the underlying gRPC code to distinguish fatal vs transient causes"],"tags":["csi","nomad","grpc","volume-delete"],"backgroundTag":"csi-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}