{"record":{"id":"46999dfa6b53d92c","repo":"hashicorp/nomad","slug":"s-error-w","errorCode":null,"errorMessage":"%s error: %w","messagePattern":"(.+?) error: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":278,"sourceCode":"\tsnap, err := a.srv.State().Snapshot()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t_, err = getNodeForRpc(snap, nodeID)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Get the connection to the client\n\tstate, ok := a.srv.getNodeConn(nodeID)\n\tif !ok {\n\t\treturn findNodeConnAndForward(a.srv, nodeID, fwdMethod, args, reply)\n\t}\n\n\t// Make the RPC\n\tif err := NodeRpc(state.Session, method, args, reply); err != nil {\n\t\treturn fmt.Errorf(\"%s error: %w\", method, err)\n\t}\n\treturn nil\n}\n\n// clientIDsForController returns a sorted list of client IDs where the\n// controller plugin is expected to be running.\nfunc (a *ClientCSI) clientIDsForController(pluginID string) ([]string, error) {\n\n\tsnap, err := a.srv.State().Snapshot()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif pluginID == \"\" {\n\t\treturn nil, fmt.Errorf(\"missing plugin ID\")\n\t}\n\n\tws := memdb.NewWatchSet()","sourceCodeStart":260,"sourceCodeEnd":296,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L260-L296","documentation":"In sendCSINodeRPC (nomad/client_csi_endpoint.go), after locating the client session for a node plugin, Nomad issues the node-scoped CSI RPC via NodeRpc; any failure is wrapped as '<method> error: %w' (e.g. CSI.NodeDetachVolume error: ...). This means the RPC reached the client/node plugin but the call itself failed, or the session transport broke.","triggerScenarios":"Called by NodeDetachVolume and NodeExpandVolume when the client's ClientCSI.NodeDetachVolume/NodeExpandVolume handler or the underlying CSI plugin returns an error, or the server-to-client RPC fails mid-call.","commonSituations":"Volume/staging claim still in use by an allocation preventing detach; node agent (nomad client) restarting during the call; CSI node plugin returning driver errors (device busy, unsupported expand); RPC timeouts.","solutions":["Read the wrapped '%w' root cause in the error to identify whether it came from the plugin or the transport.","Ensure no allocation still uses the volume (`nomad volume status <vol>`) before detaching.","Restart the nomad client agent / check client-server connectivity if the transport failed.","Check the CSI node plugin task logs on the node for driver-side errors and retry."],"exampleFix":"// before: detach while alloc still mounts volume -> driver refuses\nnomad volume detach i-abc123 ebs-vol\n// after: stop allocs using the volume first\nnomad job stop app-using-vol\nnomad volume detach i-abc123 ebs-vol","handlingStrategy":"retry","validationCode":"// ensure no allocation is using the volume before node RPCs like detach\nv, _, _ := client.Volumes().Info(ctx, volID, nil)\nif len(v.Allocs) > 0 { return fmt.Errorf(\"volume still in use by %d allocs\", len(v.Allocs)) }","typeGuard":"func nodeInUse(v *api.CSIVolume) bool { return len(v.Allocs) > 0 }","tryCatchPattern":"err := nodeRPC(args, reply)\nif err != nil {\n    var root = errors.Unwrap(err) // '%w' wraps the underlying cause\n    if isTransportError(root) { /* retry after client reconnects */ }\n    return err\n}","preventionTips":["Drain/stop workloads using the volume before detach/expand.","Unwrap the error ('%w') to read the true root cause.","Check nomad client agent health on the target node.","Retry transport-level failures, fail fast on driver rejections."],"tags":["csi","nomad","node-rpc","rpc-forwarding"],"backgroundTag":"csi-node-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}