{"record":{"id":"c4b92fc180d64029","repo":"hashicorp/nomad","slug":"csi-controllerdeletevolume-w-v","errorCode":null,"errorMessage":"CSI.ControllerDeleteVolume: %w: %v","messagePattern":"CSI\\.ControllerDeleteVolume: %w: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":284,"sourceCode":"\t\treturn fmt.Errorf(\"CSI.ControllerExpandVolume: %v\", err)\n\t}\n\tif cresp == nil {\n\t\tc.c.logger.Warn(\"plugin did not return error or response; this is a bug in the plugin and should be reported to the plugin author\")\n\t\treturn fmt.Errorf(\"CSI.ControllerExpandVolume: plugin did not return error or response\")\n\t}\n\tresp.CapacityBytes = cresp.CapacityBytes\n\tresp.NodeExpansionRequired = cresp.NodeExpansionRequired\n\treturn nil\n}\n\nfunc (c *CSI) ControllerDeleteVolume(req *structs.ClientCSIControllerDeleteVolumeRequest, resp *structs.ClientCSIControllerDeleteVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"delete_volume\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerDeleteVolume: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq := req.ToCSIRequest()\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerDeleteVolume errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\terr = plugin.ControllerDeleteVolume(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {\n\t\t// if the volume was deleted out-of-band, we'll get an error from\n\t\t// the plugin but can safely ignore it","sourceCodeStart":266,"sourceCodeEnd":302,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L266-L302","documentation":"Nomad's ControllerDeleteVolume endpoint could not find a healthy controller plugin instance matching req.PluginID (findControllerPlugin failed). The error is wrapped with structs.ErrCSIClientRPCRetryable so the Nomad server knows the plugin health view may be stale and will retry the RPC, possibly against another controller instance.","triggerScenarios":"Calling volume deletion (nomad volume delete / deregister flow) while the plugin's controller is not running, has failed its health check, has just been restarted, or the plugin ID in the request does not match any controller on the client.","commonSituations":"CSI plugin task crashed or is still starting up when a job/volume cleanup runs; node was drained or the plugin was upgraded; typo'd or stale plugin ID in the volume specification; Nomad server's catalog is out of sync after client restart.","solutions":["Retry the delete — the error is explicitly retryable and Nomad will reschedule it once a healthy controller is available","Verify the CSI plugin controller task is running and healthy (nomad job status <plugin-job>; nomad node status -verbose)","Confirm the plugin_id in the volume spec matches the controller plugin's ID (nomad plugin status <id>)","If the plugin is gone permanently, re-register the plugin job or deregister the stale volume"],"exampleFix":"// before: volume spec pointing at stale plugin\nplugin_id = \"aws-ebs-controller-old\"\n// after\nplugin_id = \"aws-ebs-controller\"","handlingStrategy":"retry","validationCode":"plug, err := apiClient.Plugins().Get(\"aws-ebs-controller\")\nif err != nil {\n    return fmt.Errorf(\"plugin not registered: %w\", err)\n}\nif plug.ControllersHealthy < 1 {\n    return fmt.Errorf(\"no healthy controller instances for %s\", plug.ID)\n}","typeGuard":"func isCSIRetryable(err error) bool {\n    return err != nil && errors.Is(err, structs.ErrCSIClientRPCRetryable)\n}","tryCatchPattern":"err := client.CSI().ControllerDeleteVolume(req)\nif errors.Is(err, structs.ErrCSIClientRPCRetryable) {\n    // wait for the server to re-route to another controller instance\n    time.Sleep(2 * time.Second)\n    return retryDelete(req)\n}\nreturn err","preventionTips":["Run the controller plugin with a restart stanza so it recovers automatically","Verify plugin_id in the volume spec matches the controller plugin's registered ID","Check nomad plugin status before issuing volume deletions","Avoid draining/upgrading the node hosting the controller during volume operations"],"tags":["csi","nomad","plugin-not-found","retryable"],"backgroundTag":"csi-plugin-unavailable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}