hashicorp/nomad · error

%s error: %w

Error message

%s error: %w

What it means

In sendCSINodeRPC (nomad/client_csi_endpoint.go), after locating the client session for a node plugin, Nomad issues the node-scoped CSI RPC via NodeRpc; any failure is wrapped as '<method> error: %w' (e.g. CSI.NodeDetachVolume error: ...). This means the RPC reached the client/node plugin but the call itself failed, or the session transport broke.

Source

Thrown at nomad/client_csi_endpoint.go:278

	snap, err := a.srv.State().Snapshot()
	if err != nil {
		return err
	}

	_, err = getNodeForRpc(snap, nodeID)
	if err != nil {
		return err
	}

	// Get the connection to the client
	state, ok := a.srv.getNodeConn(nodeID)
	if !ok {
		return findNodeConnAndForward(a.srv, nodeID, fwdMethod, args, reply)
	}

	// Make the RPC
	if err := NodeRpc(state.Session, method, args, reply); err != nil {
		return fmt.Errorf("%s error: %w", method, err)
	}
	return nil
}

// clientIDsForController returns a sorted list of client IDs where the
// controller plugin is expected to be running.
func (a *ClientCSI) clientIDsForController(pluginID string) ([]string, error) {

	snap, err := a.srv.State().Snapshot()
	if err != nil {
		return nil, err
	}

	if pluginID == "" {
		return nil, fmt.Errorf("missing plugin ID")
	}

	ws := memdb.NewWatchSet()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped '%w' root cause in the error to identify whether it came from the plugin or the transport.
2. Ensure no allocation still uses the volume (`nomad volume status <vol>`) before detaching.
3. Restart the nomad client agent / check client-server connectivity if the transport failed.
4. Check the CSI node plugin task logs on the node for driver-side errors and retry.

Example fix

// before: detach while alloc still mounts volume -> driver refuses
nomad volume detach i-abc123 ebs-vol
// after: stop allocs using the volume first
nomad job stop app-using-vol
nomad volume detach i-abc123 ebs-vol

// ensure no allocation is using the volume before node RPCs like detach
v, _, _ := client.Volumes().Info(ctx, volID, nil)
if len(v.Allocs) > 0 { return fmt.Errorf("volume still in use by %d allocs", len(v.Allocs)) }

func nodeInUse(v *api.CSIVolume) bool { return len(v.Allocs) > 0 }

err := nodeRPC(args, reply)
if err != nil {
    var root = errors.Unwrap(err) // '%w' wraps the underlying cause
    if isTransportError(root) { /* retry after client reconnects */ }
    return err
}

When it happens

Trigger: Called by NodeDetachVolume and NodeExpandVolume when the client's ClientCSI.NodeDetachVolume/NodeExpandVolume handler or the underlying CSI plugin returns an error, or the server-to-client RPC fails mid-call.

Common situations: Volume/staging claim still in use by an allocation preventing detach; node agent (nomad client) restarting during the call; CSI node plugin returning driver errors (device busy, unsupported expand); RPC timeouts.

Related errors

• controller list volumes: %v
• controller create snapshot: %v
• controller delete snapshot: %v
• controller list snapshots: %v
• unexpectedly did not forward CSIVolume.List to region %q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/46999dfa6b53d92c. Report an issue: GitHub.