{"record":{"id":"87dcd5c8fa2c9b60","repo":"hashicorp/nomad","slug":"controller-detach-volume-v","errorCode":null,"errorMessage":"controller detach volume: %v","messagePattern":"controller detach volume: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":70,"sourceCode":"\t\t\"ClientCSI.ControllerValidateVolume\",\n\t\tstructs.RateMetricWrite,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller validate volume: %v\", err)\n\t}\n\treturn nil\n}\n\nfunc (a *ClientCSI) ControllerDetachVolume(args *cstructs.ClientCSIControllerDetachVolumeRequest, reply *cstructs.ClientCSIControllerDetachVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"client_csi_controller\", \"detach_volume\"}, time.Now())\n\n\terr := a.sendCSIControllerRPC(args.PluginID,\n\t\t\"CSI.ControllerDetachVolume\",\n\t\t\"ClientCSI.ControllerDetachVolume\",\n\t\tstructs.RateMetricWrite,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller detach volume: %v\", err)\n\t}\n\treturn nil\n}\n\nfunc (a *ClientCSI) ControllerCreateVolume(args *cstructs.ClientCSIControllerCreateVolumeRequest, reply *cstructs.ClientCSIControllerCreateVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"client_csi_controller\", \"create_volume\"}, time.Now())\n\n\terr := a.sendCSIControllerRPC(args.PluginID,\n\t\t\"CSI.ControllerCreateVolume\",\n\t\t\"ClientCSI.ControllerCreateVolume\",\n\t\tstructs.RateMetricWrite,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller create volume: %v\", err)\n\t}\n\treturn nil\n}\n","sourceCodeStart":52,"sourceCodeEnd":88,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L52-L88","documentation":"ClientCSI.ControllerDetachVolume forwards a CSI ControllerDetachVolume RPC to the controller plugin and wraps failures as \"controller detach volume: <err>\". Detach runs when a volume claim is released (job stops or alloc moves), so failures here block claim unregistration.","triggerScenarios":"Job shutdown or node drain releasing a CSI volume claim when the controller plugin is unreachable, errors on the underlying CSI ControllerUnpublishVolume, or the plugin is not a controller.","commonSituations":"Controller plugin crashed after the workload ran; storage backend reporting the volume not attached / node still holding it; drain with volumes where the controller was collocated on the draining node; plugin_id mismatch.","solutions":["Restore the controller plugin (restart its task) and retry the detach / re-run `nomad volume detach` or `nomad volume claim -allow-multi`.","Check plugin logs for the CSI error; some backends need the node's unpublish to complete first (stop the workload alloc).","Use `nomad volume claim -delete <vol> <node> <alloc>` to clear stale claims so future detaches aren't attempted.","Verify the volume's plugin_id points to a healthy controller, not just a node plugin."],"exampleFix":"# before: claim stuck after plugin crash\nnomad volume status ebs-vol   # claim present, detach failing\n# after: restart controller plugin task, then clear stale claim\nnomad volume detach ebs-vol <node-id>\nnomad volume claim -delete ebs-vol <node-id> <alloc-id>","handlingStrategy":"retry","validationCode":"const plugin = await nomad.plugin(pluginID)\nif (!plugin.controllers?.some(c => c.healthy)) {\n  throw new Error('controller plugin must be healthy before releasing volume claims')\n}","typeGuard":null,"tryCatchPattern":"await withRetry(async () => {\n  try { return await releaseClaim(volumeID) }\n  catch (e) {\n    if (String(e).startsWith('controller detach volume')) throw new RetryableError(e)\n    throw e\n  }\n}, { attempts: 5, backoff: 'exponential' })","preventionTips":["Ensure the controller plugin is not scheduled only on nodes being drained.","Stop workload allocs before detaching so node-side unpublish completes first.","Clear stale claims with `nomad volume claim -delete` before retrying.","Add drain hooks that verify controller plugin health for volume-hosting nodes."],"tags":["nomad","csi","storage","detach","plugin"],"backgroundTag":"csi-controller-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}