hashicorp/nomad · warning

CSI.ControllerCreateSnapshot: %w: %v

Error message

CSI.ControllerCreateSnapshot: %w: %v

What it means

CSI.ControllerCreateSnapshot wraps the server RPC ControllerCreateSnapshot. When findControllerPlugin fails to locate a healthy controller plugin on this client, Nomad wraps the failure together with structs.ErrCSIClientRPCRetryable, telling the server that its view of plugin health/registration is stale and the RPC should be retried against another controller instance.

Source

Thrown at client/csi_endpoint.go:376

			}
		}
		resp.Entries = append(resp.Entries, vol)
		if req.MaxEntries != 0 && int32(len(resp.Entries)) == req.MaxEntries {
			break
		}
	}

	return nil
}

func (c *CSI) ControllerCreateSnapshot(req *structs.ClientCSIControllerCreateSnapshotRequest, resp *structs.ClientCSIControllerCreateSnapshotResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "create_snapshot"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerCreateSnapshot: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq, err := req.ToCSIRequest()
	if err != nil {
		return fmt.Errorf("CSI.ControllerCreateSnapshot: %v", err)
	}

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerCreateSnapshot errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	cresp, err := plugin.ControllerCreateSnapshot(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the snapshot request; ErrCSIClientRPCRetryable makes Nomad retry on another controller automatically
2. Check the plugin job is running and includes the controller plugin (`nomad job status <plugin-job>`, `nomad plugin status`)
3. Verify the volume spec's plugin_id matches the running controller plugin's ID
4. Ensure the plugin's CSI Controller service (not just Node) is served and healthy

Example fix

// before
plugin_id = "aws-ebs-node"  // only the node plugin
// after
plugin_id = "aws-ebs-controller"  // plugin serving the Controller service

// Before snapshotting, verify the controller plugin is healthy:
// nomad plugin status
// Ensure the plugin job serving the Controller service is running and stable.

// ErrCSIClientRPCRetryable means the server will re-route; callers should retry with backoff
if err := c.ControllerCreateSnapshot(req, resp); err != nil {
    if errors.Is(err, structs.ErrCSIClientRPCRetryable) {
        // retry, possibly against another client/controller
    }
}

When it happens

Trigger: req.PluginID does not match any controller plugin currently registered/healthy on this client: plugin not running, controller (vs node) plugin not started, plugin just crashed, or the server routed the RPC to a client whose registration hasn't caught up.

Common situations: Deploying a plugin job where only the node plugin came up; plugin job crashed or was updated so the controller deregistered; race right after plugin registration where server state is stale; typo'd/changed plugin ID in the volume spec.

Related errors

• CSI.ControllerDeleteSnapshot: %w: %v
• CSI.ControllerListSnapshots: %w: %v
• CSI.ControllerCreateSnapshot: plugin did not return error or
• CSI.ControllerListSnapshot: plugin returned an invalid entry
• could not query plugin %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/da458425acac82c0. Report an issue: GitHub.