{"record":{"id":"b35f94981b6a035b","repo":"hashicorp/nomad","slug":"controller-attach-volume-v","errorCode":null,"errorMessage":"controller attach volume: %v","messagePattern":"controller attach volume: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":42,"sourceCode":"\tsrv    *Server\n\tctx    *RPCContext\n\tlogger log.Logger\n}\n\nfunc NewClientCSIEndpoint(srv *Server, ctx *RPCContext) *ClientCSI {\n\treturn &ClientCSI{srv: srv, ctx: ctx, logger: srv.logger.Named(\"client_csi\")}\n}\n\nfunc (a *ClientCSI) ControllerAttachVolume(args *cstructs.ClientCSIControllerAttachVolumeRequest, reply *cstructs.ClientCSIControllerAttachVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"client_csi_controller\", \"attach_volume\"}, time.Now())\n\n\terr := a.sendCSIControllerRPC(args.PluginID,\n\t\t\"CSI.ControllerAttachVolume\",\n\t\t\"ClientCSI.ControllerAttachVolume\",\n\t\tstructs.RateMetricWrite,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller attach volume: %v\", err)\n\t}\n\treturn nil\n}\n\nfunc (a *ClientCSI) ControllerValidateVolume(args *cstructs.ClientCSIControllerValidateVolumeRequest, reply *cstructs.ClientCSIControllerValidateVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"client_csi_controller\", \"validate_volume\"}, time.Now())\n\n\terr := a.sendCSIControllerRPC(args.PluginID,\n\t\t\"CSI.ControllerValidateVolume\",\n\t\t\"ClientCSI.ControllerValidateVolume\",\n\t\tstructs.RateMetricWrite,\n\t\targs, reply)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"controller validate volume: %v\", err)\n\t}\n\treturn nil\n}\n","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L24-L60","documentation":"ClientCSI.ControllerAttachVolume is the client-agent bridge that forwards a CSI ControllerAttachVolume RPC to the CSI controller plugin on the node via sendCSIControllerRPC. Any failure in that forwarding (plugin missing, plugin RPC error, timeout) is wrapped as \"controller attach volume: <err>\" and returned to the server, which surfaces it to the volume claim workflow.","triggerScenarios":"Volume claim/unclaim (job with volume mounts) when the controller plugin with args.PluginID is not running on the target node, the plugin socket is broken, or the underlying CSI AttachVolume call fails.","commonSituations":"CSI plugin crashed or not deployed; wrong plugin_id in the volume spec; plugin lacks ControllerCapabilities (attach); storage backend rejecting the attach (volume in use, invalid volume ID); node-plugin registration incomplete.","solutions":["Check plugin health: `nomad plugin status <plugin-id>` and confirm a controller instance is healthy.","Verify the volume spec's plugin_id and volume ID match the deployed controller (`nomad volume status <vol-id>`).","Check the node's plugin logs (`nomad alloc logs <plugin-alloc-id>`) for the underlying CSI error and fix the storage-side cause.","Re-register the plugin (restart the plugin task) if the socket/handshake is broken."],"exampleFix":"// before\nvolume {\n  type = \"csi\"\n  source = \"ebs-vol\"\n}\n// after\nvolume \"data\" {\n  type = \"csi\"\n  source = \"ebs-vol\"              # must exist and match a registered volume\n  # ensure plugin_id of this volume points to a HEALTHY controller plugin\n  # nomad plugin status ebs-plugin -> ControllerHealthy = true\n}","handlingStrategy":"try-catch","validationCode":"const plugin = await nomad.plugin(pluginID)\nif (!plugin.controllers || !plugin.controllers.some(c => c.healthy)) {\n  throw new Error(`CSI controller ${pluginID} not healthy; restore before volume claims`)\n}","typeGuard":"const hasHealthyController = (p) => !!p?.controllers?.some(c => c.healthy)","tryCatchPattern":"try { await volumeClaim(volumeID) }\ncatch (e) {\n  if (String(e).startsWith('controller attach volume')) {\n    log.error('CSI controller attach failed; check plugin:', await pluginLogs(pluginID))\n    throw new RetryableStorageError(e)\n  }\n  throw e\n}","preventionTips":["Monitor `nomad plugin status` for controller health as part of cluster dashboards.","Pin plugin_id in volume specs to the deployed controller's ID.","Pre-validate volume IDs against the storage backend before claims.","Restart/unhealthy plugin alerts before batch jobs that mount volumes.","Check that plugin exposes ATTACH controller capability."],"tags":["nomad","csi","storage","plugin"],"backgroundTag":"csi-controller-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}