hashicorp/nomad · error

controller delete volume: %v

Error message

controller delete volume: %v

What it means

ClientCSI.ControllerDeleteVolume forwards a CSI ControllerDeleteVolume RPC to the controller plugin and wraps failures as "controller delete volume: <err>". It runs when a dynamically provisioned volume is deregistered with purge, asking the controller to delete the backing storage object.

Source

Thrown at nomad/client_csi_endpoint.go:112

		"ClientCSI.ControllerExpandVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller expand volume: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerDeleteVolume(args *cstructs.ClientCSIControllerDeleteVolumeRequest, reply *cstructs.ClientCSIControllerDeleteVolumeResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "delete_volume"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerDeleteVolume",
		"ClientCSI.ControllerDeleteVolume",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller delete volume: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerListVolumes(args *cstructs.ClientCSIControllerListVolumesRequest, reply *cstructs.ClientCSIControllerListVolumesResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "list_volumes"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerListVolumes",
		"ClientCSI.ControllerListVolumes",
		structs.RateMetricList,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller list volumes: %v", err)
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Release all claims first: stop jobs using the volume, then `nomad volume detach` / `nomad volume claim -delete` until no claims remain, then retry purge.
2. If the volume is external (not dynamically created), deregister WITHOUT -purge; deletion is not applicable.
3. Restart/repair the controller plugin and check its logs for the backend deletion error; resolve storage-side dependencies (snapshots, attachments) then retry.

Example fix

# before: purge fails because claims remain
nomad volume deregister -purge ebs-vol
# after
nomad volume detach ebs-vol <node-id>
nomad volume claim -delete ebs-vol <node-id> <alloc-id>
nomad volume deregister -purge ebs-vol

const vol = await nomad.volume(volumeID)
const outstanding = await nomad.volumeClaims(volumeID)
if (vol.externalId && !vol.dynamicallyCreated) {
  throw new Error('external volume: use deregister without -purge; the controller will not delete it')
}
if (outstanding.length > 0) {
  throw new Error(`release ${outstanding.length} claim(s) before purging`)
}

const canPurge = (vol, claims) => vol.dynamicallyCreated && claims.length === 0

try { await deregisterVolume(id, { purge: true }) }
catch (e) {
  if (String(e).startsWith('controller delete volume')) {
    await releaseAllClaims(id)
    return deregisterVolume(id, { purge: true }) // retry once after cleanup
  }
  throw e
}

When it happens

Trigger: `nomad volume deregister -purge` (or deregistration of a dynamic volume) when the controller plugin is unreachable or the backend refuses deletion (volume still attached, snapshots exist, volume is external/not created by the plugin).

Common situations: Claims still outstanding so the backend keeps the volume; attempting to purge an externally pre-provisioned volume the controller won't delete; controller plugin down; storage-side dependency (snapshot, clone).

Related errors

• controller attach volume: %v
• controller validate volume: %v
• controller detach volume: %v
• controller create volume: %v
• controller expand volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8192d828b9fe2099. Report an issue: GitHub.