hashicorp/nomad · warning

CSI.ControllerListSnapshots: %w: %v

Error message

CSI.ControllerListSnapshots: %w: %v

What it means

CSI.ControllerListSnapshots wraps the server RPC ControllerListSnapshots. As with other controller RPCs, a failure from findControllerPlugin (no healthy controller plugin for req.PluginID on this client) is wrapped with structs.ErrCSIClientRPCRetryable so the server knows to retry with another controller instance.

Source

Thrown at client/csi_endpoint.go:454

		// if the snapshot was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it
		c.c.logger.Debug("could not delete snapshot", "error", err)
		return nil
	}
	if err != nil {
		return fmt.Errorf("CSI.ControllerDeleteSnapshot: %v", err)
	}
	return err
}

func (c *CSI) ControllerListSnapshots(req *structs.ClientCSIControllerListSnapshotsRequest, resp *structs.ClientCSIControllerListSnapshotsResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "list_snapshots"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerListSnapshots: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerListSnapshots errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	cresp, err := plugin.ControllerListSnapshots(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if err != nil {
		return fmt.Errorf("CSI.ControllerListSnapshots: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry; the RPC will be routed to another healthy controller
2. Confirm a healthy controller plugin via `nomad plugin status` and the plugin job status
3. Correct the plugin_id used for snapshot listing queries
4. Ensure the controller plugin serves the CSI Controller service and is not stuck in pending

Example fix

# before
$ nomad snapshot list -plugin broken-id
# after
$ nomad plugin status   # find the healthy plugin id first
$ nomad snapshot list -plugin healthy-controller-id

// Verify controller health before listing:
// nomad plugin status
// Restart/reroute if the target controller plugin shows unhealthy.

// Retryable marker: server will retry on another controller
if err := c.ControllerListSnapshots(req, resp); err != nil {
    if errors.Is(err, structs.ErrCSIClientRPCRetryable) {
        // retry with backoff
    }
}

When it happens

Trigger: ListSnapshots arrives at a client lacking a registered healthy controller for req.PluginID — plugin crashed, only node plugin running, or registration is stale.

Common situations: Listing snapshots while the controller plugin is restarting during an update; multi-client fleets where only some clients run the controller; plugin ID mismatches after job rename.

Related errors

• CSI.ControllerCreateSnapshot: %w: %v
• CSI.ControllerDeleteSnapshot: %w: %v
• CSI.ControllerCreateSnapshot: plugin did not return error or
• CSI.ControllerListSnapshot: plugin returned an invalid entry
• could not query plugin %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f06898e29efc7a80. Report an issue: GitHub.