hashicorp/nomad · error

controller attach volume: %v

Error message

controller attach volume: %v

What it means

ClientCSI.ControllerAttachVolume is the client-agent bridge that forwards a CSI ControllerAttachVolume RPC to the CSI controller plugin on the node via sendCSIControllerRPC. Any failure in that forwarding (plugin missing, plugin RPC error, timeout) is wrapped as "controller attach volume: <err>" and returned to the server, which surfaces it to the volume claim workflow.

Source

Thrown at nomad/client_csi_endpoint.go:42

	srv    *Server
	ctx    *RPCContext
	logger log.Logger
}

func NewClientCSIEndpoint(srv *Server, ctx *RPCContext) *ClientCSI {
	return &ClientCSI{srv: srv, ctx: ctx, logger: srv.logger.Named("client_csi")}
}

func (a *ClientCSI) ControllerAttachVolume(args *cstructs.ClientCSIControllerAttachVolumeRequest, reply *cstructs.ClientCSIControllerAttachVolumeResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "attach_volume"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerAttachVolume",
		"ClientCSI.ControllerAttachVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller attach volume: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerValidateVolume(args *cstructs.ClientCSIControllerValidateVolumeRequest, reply *cstructs.ClientCSIControllerValidateVolumeResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "validate_volume"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerValidateVolume",
		"ClientCSI.ControllerValidateVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller validate volume: %v", err)
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check plugin health: `nomad plugin status <plugin-id>` and confirm a controller instance is healthy.
2. Verify the volume spec's plugin_id and volume ID match the deployed controller (`nomad volume status <vol-id>`).
3. Check the node's plugin logs (`nomad alloc logs <plugin-alloc-id>`) for the underlying CSI error and fix the storage-side cause.
4. Re-register the plugin (restart the plugin task) if the socket/handshake is broken.

Example fix

// before
volume {
  type = "csi"
  source = "ebs-vol"
}
// after
volume "data" {
  type = "csi"
  source = "ebs-vol"              # must exist and match a registered volume
  # ensure plugin_id of this volume points to a HEALTHY controller plugin
  # nomad plugin status ebs-plugin -> ControllerHealthy = true
}

const plugin = await nomad.plugin(pluginID)
if (!plugin.controllers || !plugin.controllers.some(c => c.healthy)) {
  throw new Error(`CSI controller ${pluginID} not healthy; restore before volume claims`)
}

const hasHealthyController = (p) => !!p?.controllers?.some(c => c.healthy)

try { await volumeClaim(volumeID) }
catch (e) {
  if (String(e).startsWith('controller attach volume')) {
    log.error('CSI controller attach failed; check plugin:', await pluginLogs(pluginID))
    throw new RetryableStorageError(e)
  }
  throw e
}

When it happens

Trigger: Volume claim/unclaim (job with volume mounts) when the controller plugin with args.PluginID is not running on the target node, the plugin socket is broken, or the underlying CSI AttachVolume call fails.

Common situations: CSI plugin crashed or not deployed; wrong plugin_id in the volume spec; plugin lacks ControllerCapabilities (attach); storage backend rejecting the attach (volume in use, invalid volume ID); node-plugin registration incomplete.

Related errors

• controller validate volume: %v
• controller detach volume: %v
• controller create volume: %v
• controller expand volume: %v
• controller delete volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b35f94981b6a035b. Report an issue: GitHub.