hashicorp/nomad · error

controller detach volume: %v

Error message

controller detach volume: %v

What it means

ClientCSI.ControllerDetachVolume forwards a CSI ControllerDetachVolume RPC to the controller plugin and wraps failures as "controller detach volume: <err>". Detach runs when a volume claim is released (job stops or alloc moves), so failures here block claim unregistration.

Source

Thrown at nomad/client_csi_endpoint.go:70

		"ClientCSI.ControllerValidateVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller validate volume: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerDetachVolume(args *cstructs.ClientCSIControllerDetachVolumeRequest, reply *cstructs.ClientCSIControllerDetachVolumeResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "detach_volume"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerDetachVolume",
		"ClientCSI.ControllerDetachVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller detach volume: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerCreateVolume(args *cstructs.ClientCSIControllerCreateVolumeRequest, reply *cstructs.ClientCSIControllerCreateVolumeResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "create_volume"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerCreateVolume",
		"ClientCSI.ControllerCreateVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller create volume: %v", err)
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restore the controller plugin (restart its task) and retry the detach / re-run `nomad volume detach` or `nomad volume claim -allow-multi`.
2. Check plugin logs for the CSI error; some backends need the node's unpublish to complete first (stop the workload alloc).
3. Use `nomad volume claim -delete <vol> <node> <alloc>` to clear stale claims so future detaches aren't attempted.
4. Verify the volume's plugin_id points to a healthy controller, not just a node plugin.

Example fix

# before: claim stuck after plugin crash
nomad volume status ebs-vol   # claim present, detach failing
# after: restart controller plugin task, then clear stale claim
nomad volume detach ebs-vol <node-id>
nomad volume claim -delete ebs-vol <node-id> <alloc-id>

const plugin = await nomad.plugin(pluginID)
if (!plugin.controllers?.some(c => c.healthy)) {
  throw new Error('controller plugin must be healthy before releasing volume claims')
}

await withRetry(async () => {
  try { return await releaseClaim(volumeID) }
  catch (e) {
    if (String(e).startsWith('controller detach volume')) throw new RetryableError(e)
    throw e
  }
}, { attempts: 5, backoff: 'exponential' })

When it happens

Trigger: Job shutdown or node drain releasing a CSI volume claim when the controller plugin is unreachable, errors on the underlying CSI ControllerUnpublishVolume, or the plugin is not a controller.

Common situations: Controller plugin crashed after the workload ran; storage backend reporting the volume not attached / node still holding it; drain with volumes where the controller was collocated on the draining node; plugin_id mismatch.

Related errors

• controller attach volume: %v
• controller validate volume: %v
• controller create volume: %v
• controller expand volume: %v
• controller delete volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/87dcd5c8fa2c9b60. Report an issue: GitHub.