hashicorp/nomad · warning

CSI.ControllerDeleteSnapshot: %w: %v

Error message

CSI.ControllerDeleteSnapshot: %w: %v

What it means

CSI.ControllerDeleteSnapshot wraps the server RPC ControllerDeleteSnapshot. Like the create path, if findControllerPlugin cannot find a healthy controller plugin for req.PluginID on this client, the error is wrapped with structs.ErrCSIClientRPCRetryable so the server retries the deletion against a different controller instance.

Source

Thrown at client/csi_endpoint.go:419

		return fmt.Errorf("CSI.ControllerCreateSnapshot: plugin did not return error or snapshot")
	}
	resp.ID = cresp.Snapshot.ID
	resp.ExternalSourceVolumeID = cresp.Snapshot.SourceVolumeID
	resp.SizeBytes = cresp.Snapshot.SizeBytes
	resp.CreateTime = cresp.Snapshot.CreateTime
	resp.IsReady = cresp.Snapshot.IsReady

	return nil
}

func (c *CSI) ControllerDeleteSnapshot(req *structs.ClientCSIControllerDeleteSnapshotRequest, resp *structs.ClientCSIControllerDeleteSnapshotResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "delete_snapshot"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerDeleteSnapshot: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerDeleteSnapshot errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	err = plugin.ControllerDeleteSnapshot(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the snapshot was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Let Nomad retry — the retryable marker routes the RPC to another healthy controller
2. Restore/restart the controller plugin job (`nomad job start`/`nomad job revert`)
3. Verify with `nomad plugin status` that a controller plugin for the storage provider is healthy
4. Re-run the snapshot deletion once the plugin re-registers

Example fix

// before
plugin_id = "ebs-controller-v1"  // job updated, plugin now registers as v2
// after
plugin_id = "ebs-controller-v2"

// Confirm a healthy controller exists before deleting snapshots:
// nomad plugin status
// and check the plugin job: nomad job status <csi-plugin-job>

// Retryable: deletion will be re-routed to another controller
if err := c.ControllerDeleteSnapshot(req, resp); err != nil {
    if errors.Is(err, structs.ErrCSIClientRPCRetryable) {
        // retry / wait for plugin re-registration
    }
}

When it happens

Trigger: DeleteSnapshot routed to a client where the named controller plugin is absent, deregistered, crashed, or still registering; plugin ID mismatch between the snapshot/volume registration and the running plugin job.

Common situations: Controller plugin job stopped/failed before cleanup ran; node drains or client restarts leaving stale plugin registrations; plugin updated with a new ID; snapshot garbage collection racing plugin shutdown.

Related errors

• CSI.ControllerCreateSnapshot: %w: %v
• CSI.ControllerListSnapshots: %w: %v
• CSI.ControllerCreateSnapshot: plugin did not return error or
• CSI.ControllerListSnapshot: plugin returned an invalid entry
• could not query plugin %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2b81745716ccbe62. Report an issue: GitHub.