hashicorp/nomad · error

CSI.ControllerDeleteVolume: %v

Error message

CSI.ControllerDeleteVolume: %v

What it means

The CSI plugin's ControllerDeleteVolume RPC returned an error that is neither a NOT_FOUND-style code (which would be logged as an out-of-band deletion and ignored) nor retried out; Nomad surfaces it wrapped as 'CSI.ControllerDeleteVolume: <err>'. It means the plugin itself refused or failed the delete operation.

Source

Thrown at client/csi_endpoint.go:307

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerDeleteVolume errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	err = plugin.ControllerDeleteVolume(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the volume was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it
		c.c.logger.Debug("could not delete volume", "error", err)
		return nil
	}
	if err != nil {
		return fmt.Errorf("CSI.ControllerDeleteVolume: %v", err)
	}
	return err
}

func (c *CSI) ControllerListVolumes(req *structs.ClientCSIControllerListVolumesRequest, resp *structs.ClientCSIControllerListVolumesResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "list_volumes"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerListVolumes: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped plugin error and the plugin logs to identify the underlying storage-backend failure
2. Ensure the volume is not still attached/mounted (check nomad volume status, detach allocations or wait for node unpublish)
3. Verify the volume's external/source ID exists in the storage backend; if it was already deleted out-of-band, the volume can safely be deregistered from Nomad
4. Fix backend-side issues (credentials, quotas, in-use snapshots) and retry the delete

Example fix

// before: deleting while volume still in use
nomad volume delete -volume-id mysql-data
// after: stop allocations using it first
nomad job stop mysql
nomad volume detach -node-id <node> -volume-id mysql-data
nomad volume delete -volume-id mysql-data

vols, _, err := apiClient.Volumes().List(nil)
if err == nil {
    for _, v := range vols {
        if v.ID == volID && v.Schedulable && v.ControllersHealthy > 0 {
            // check no live allocations claim it
            detail, _, err := apiClient.Volumes().Get(volID, nil)
            if err == nil && len(detail.Allocations) > 0 {
                return fmt.Errorf("volume %s still has allocations; stop jobs before delete", volID)
            }
        }
    }
}

func isVolumeInUseErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "FailedPrecondition")
}

err := client.CSI().ControllerDeleteVolume(req)
if err != nil {
    var st *status.Status
    if errors.As(err, &st) && st.Code() == codes.NotFound {
        // volume already deleted out-of-band: safe to ignore/deregister
        return nil
    }
    if isVolumeInUseErr(err) {
        return fmt.Errorf("detach volume from all nodes before delete: %w", err)
    }
    return err
}

When it happens

Trigger: The plugin returns codes other than NotFound/ignorable ones — e.g. Internal errors from the storage backend, FailedPrecondition when the volume is in use, InvalidArgument for a bad external volume ID, or transient gRPC failures after the 3 retries with backoff are exhausted.

Common situations: Volume still attached/mounted by nodes when delete is issued; underlying storage backend (EBS, Ceph, etc.) rejects deletion; volume was created outside Nomad so the external ID does not exist in the backend; permission/credential problems for the plugin's cloud account.

Related errors

• CSI plugin failed to register: %w
• failed to probe plugin: %w
• CSI.ControllerExpandVolume: plugin did not return error or r
• CSI.ControllerListVolumes: %v
• %v: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4213eca757c16088. Report an issue: GitHub.