{"record":{"id":"da458425acac82c0","repo":"hashicorp/nomad","slug":"csi-controllercreatesnapshot-w-v","errorCode":null,"errorMessage":"CSI.ControllerCreateSnapshot: %w: %v","messagePattern":"CSI\\.ControllerCreateSnapshot: %w: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/csi_endpoint.go","lineNumber":376,"sourceCode":"\t\t\t}\n\t\t}\n\t\tresp.Entries = append(resp.Entries, vol)\n\t\tif req.MaxEntries != 0 && int32(len(resp.Entries)) == req.MaxEntries {\n\t\t\tbreak\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc (c *CSI) ControllerCreateSnapshot(req *structs.ClientCSIControllerCreateSnapshotRequest, resp *structs.ClientCSIControllerCreateSnapshotResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"create_snapshot\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerCreateSnapshot: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq, err := req.ToCSIRequest()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerCreateSnapshot: %v\", err)\n\t}\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerCreateSnapshot errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\tcresp, err := plugin.ControllerCreateSnapshot(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))","sourceCodeStart":358,"sourceCodeEnd":394,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L358-L394","documentation":"CSI.ControllerCreateSnapshot wraps the server RPC ControllerCreateSnapshot. When findControllerPlugin fails to locate a healthy controller plugin on this client, Nomad wraps the failure together with structs.ErrCSIClientRPCRetryable, telling the server that its view of plugin health/registration is stale and the RPC should be retried against another controller instance.","triggerScenarios":"req.PluginID does not match any controller plugin currently registered/healthy on this client: plugin not running, controller (vs node) plugin not started, plugin just crashed, or the server routed the RPC to a client whose registration hasn't caught up.","commonSituations":"Deploying a plugin job where only the node plugin came up; plugin job crashed or was updated so the controller deregistered; race right after plugin registration where server state is stale; typo'd/changed plugin ID in the volume spec.","solutions":["Retry the snapshot request; ErrCSIClientRPCRetryable makes Nomad retry on another controller automatically","Check the plugin job is running and includes the controller plugin (`nomad job status <plugin-job>`, `nomad plugin status`)","Verify the volume spec's plugin_id matches the running controller plugin's ID","Ensure the plugin's CSI Controller service (not just Node) is served and healthy"],"exampleFix":"// before\nplugin_id = \"aws-ebs-node\"  // only the node plugin\n// after\nplugin_id = \"aws-ebs-controller\"  // plugin serving the Controller service","handlingStrategy":"retry","validationCode":"// Before snapshotting, verify the controller plugin is healthy:\n// nomad plugin status\n// Ensure the plugin job serving the Controller service is running and stable.","typeGuard":null,"tryCatchPattern":"// ErrCSIClientRPCRetryable means the server will re-route; callers should retry with backoff\nif err := c.ControllerCreateSnapshot(req, resp); err != nil {\n    if errors.Is(err, structs.ErrCSIClientRPCRetryable) {\n        // retry, possibly against another client/controller\n    }\n}","preventionTips":["Always run the controller plugin job alongside the node plugin","Verify plugin_id in volume specs matches the running plugin","Wait for plugin registration before issuing snapshot RPCs","Watch `nomad plugin status` for deregistrations after job updates"],"tags":["csi","snapshot","plugin","retry"],"backgroundTag":"csi-plugin-not-found-retryable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}