{"record":{"id":"39d0f57dd763999c","repo":"hashicorp/nomad","slug":"csi-controllerdetachvolume-w-v-wraps-errcsicl","errorCode":null,"errorMessage":"CSI.ControllerDetachVolume: %w: %v (wraps ErrCSIClientRPCRetryable)","messagePattern":"CSI\\.ControllerDetachVolume: %w: (.+?) \\(wraps ErrCSIClientRPCRetryable\\)","errorType":"exception","errorClass":"ErrCSIClientRPCRetryable","httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":144,"sourceCode":"\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerAttachVolume: %v\", err)\n\t}\n\n\tresp.PublishContext = cresp.PublishContext\n\treturn nil\n}\n\n// ControllerDetachVolume is used to detach a volume from a CSI Cluster from\n// the storage node provided in the request.\nfunc (c *CSI) ControllerDetachVolume(req *structs.ClientCSIControllerDetachVolumeRequest, resp *structs.ClientCSIControllerDetachVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"unpublish_volume\"}, time.Now())\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerDetachVolume: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\t// The following block of validation checks should not be reached on a\n\t// real Nomad cluster as all of this data should be validated when registering\n\t// volumes with the cluster. They serve as a defensive check before forwarding\n\t// requests to plugins, and to aid with development.\n\n\tif req.VolumeID == \"\" {\n\t\treturn errors.New(\"CSI.ControllerDetachVolume: VolumeID is required\")\n\t}\n\n\tif req.ClientCSINodeID == \"\" {\n\t\treturn errors.New(\"CSI.ControllerDetachVolume: ClientCSINodeID is required\")\n\t}\n\n\tcsiReq := req.ToCSIRequest()","sourceCodeStart":126,"sourceCodeEnd":162,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L126-L162","documentation":"ControllerDetachVolume wraps a findControllerPlugin failure in ErrCSIClientRPCRetryable so the server knows the plugin-health view is stale and can retry with another controller instance. The cause is the absence of a healthy controller plugin for req.PluginID on this client.","triggerScenarios":"An unpublish/detach RPC arrives while the controller plugin is deregistered, unhealthy, or scheduled on a different client than the one handling req.PluginID.","commonSituations":"Controller plugin crashed between attach and detach (leaving a stale attachment); plugin job stopped during maintenance; server retrying unpublish against a node where the plugin no longer runs.","solutions":["Restore/restart the controller plugin job on the cluster so findControllerPlugin can locate it","Retry the detach — the retryable sentinel tells the server to attempt another controller instance","If the attachment is orphaned, use the provider CLI or `nomad volume detach` with the correct node/plugin to force detach","Verify plugin ID consistency between volume registration and the running plugin job"],"exampleFix":"// before\nnomad job stop efs-controller   # detach now fails with retryable plugin error\n// after\nnomad job start efs-controller && nomad volume detach <vol> <node>","handlingStrategy":"retry","validationCode":"// before detach, verify controller plugin availability\nif p := clientCSIPlugin(pluginID); p == nil {\n    return structs.NewErrRPCCallFailed(clientAddr, \"controller plugin gone; retry elsewhere\")\n}","typeGuard":"func isRetryableCSI(err error) bool { return structs.IsErrRetryable(err) }","tryCatchPattern":"err := c.ControllerDetachVolume(req, resp)\nif err != nil {\n    if structs.IsErrRetryable(err) {\n        // re-route to another healthy controller\n        return structs.NewErrRPCCallFailed(addr, err.Error())\n    }\n    return err\n}","preventionTips":["Avoid stopping controller plugin jobs while volumes are claimed","Use `nomad volume status` to confirm which node/plugin owns attachments","Keep controller plugins redundant across nodes where the driver allows","Automate plugin health alerts before maintenance windows"],"tags":["csi","plugin","retryable","detach"],"backgroundTag":"csi-plugin-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}