hashicorp/nomad · error

CSI.ControllerDeleteSnapshot: %v

Error message

CSI.ControllerDeleteSnapshot: %v

What it means

CSI.ControllerDeleteSnapshot wraps any non-ignored error from the plugin's ControllerDeleteSnapshot gRPC call with this message. Note the preceding branch: if the plugin reports the snapshot was already deleted out-of-band, Nomad logs at debug and returns nil. So this error means a genuine deletion failure that is not 'snapshot already gone'.

Source

Thrown at client/csi_endpoint.go:442

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerDeleteSnapshot errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	err = plugin.ControllerDeleteSnapshot(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the snapshot was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it
		c.c.logger.Debug("could not delete snapshot", "error", err)
		return nil
	}
	if err != nil {
		return fmt.Errorf("CSI.ControllerDeleteSnapshot: %v", err)
	}
	return err
}

func (c *CSI) ControllerListSnapshots(req *structs.ClientCSIControllerListSnapshotsRequest, resp *structs.ClientCSIControllerListSnapshotsResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "list_snapshots"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerListSnapshots: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped gRPC/CSI status in the message to see the backend's actual complaint
2. Verify the snapshot ID exists in the storage backend; if already gone, retry — Nomad ignores not-found
3. Check plugin credentials/permissions to delete snapshots in the backend account
4. Check plugin logs and backend status, then retry the delete

Example fix

// before (plugin side): returning generic error when snapshot missing
return nil, status.Error(codes.Internal, "snapshot missing")
// after
return nil, status.Error(codes.NotFound, "snapshot already deleted")

// Check the snapshot still exists before deleting:
// nomad volume snapshot list -plugin <plugin-id>
// Only issue DeleteSnapshot for snapshots present in the listing.

// Distinguish not-found (ignored by Nomad) from real failures
err := c.ControllerDeleteSnapshot(req, resp)
if err != nil {
    if strings.Contains(err.Error(), "CSI.ControllerDeleteSnapshot:") {
        // inspect wrapped gRPC/CSI status; treat NotFound as already-deleted success
    }
}

When it happens

Trigger: plugin.ControllerDeleteSnapshot returns an error that is not the 'not found'/'deleted out-of-band' case: invalid snapshot ID, permissions failure, backend outage, or fatal CSI status code.

Common situations: Backend removed the snapshot and the plugin surfaces a non-standard error code Nomad doesn't recognize as not-found; IAM/credentials revoked between create and delete; storage backend API outage; deleting snapshots while the plugin is being upgraded.

Related errors

• CSI.ControllerListSnapshots: %v
• CSI.ControllerListSnapshot: plugin returned an invalid entry
• snapshot %q already exists but is incompatible with volume I
• snapshot %q is already pending: %v
• storage provider does not have enough space for this snapsho

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/eaae17c4c08d887b. Report an issue: GitHub.