{"record":{"id":"2b81745716ccbe62","repo":"hashicorp/nomad","slug":"csi-controllerdeletesnapshot-w-v","errorCode":null,"errorMessage":"CSI.ControllerDeleteSnapshot: %w: %v","messagePattern":"CSI\\.ControllerDeleteSnapshot: %w: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/csi_endpoint.go","lineNumber":419,"sourceCode":"\t\treturn fmt.Errorf(\"CSI.ControllerCreateSnapshot: plugin did not return error or snapshot\")\n\t}\n\tresp.ID = cresp.Snapshot.ID\n\tresp.ExternalSourceVolumeID = cresp.Snapshot.SourceVolumeID\n\tresp.SizeBytes = cresp.Snapshot.SizeBytes\n\tresp.CreateTime = cresp.Snapshot.CreateTime\n\tresp.IsReady = cresp.Snapshot.IsReady\n\n\treturn nil\n}\n\nfunc (c *CSI) ControllerDeleteSnapshot(req *structs.ClientCSIControllerDeleteSnapshotRequest, resp *structs.ClientCSIControllerDeleteSnapshotResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"delete_snapshot\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerDeleteSnapshot: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq := req.ToCSIRequest()\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerDeleteSnapshot errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\terr = plugin.ControllerDeleteSnapshot(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {\n\t\t// if the snapshot was deleted out-of-band, we'll get an error from\n\t\t// the plugin but can safely ignore it","sourceCodeStart":401,"sourceCodeEnd":437,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L401-L437","documentation":"CSI.ControllerDeleteSnapshot wraps the server RPC ControllerDeleteSnapshot. Like the create path, if findControllerPlugin cannot find a healthy controller plugin for req.PluginID on this client, the error is wrapped with structs.ErrCSIClientRPCRetryable so the server retries the deletion against a different controller instance.","triggerScenarios":"DeleteSnapshot routed to a client where the named controller plugin is absent, deregistered, crashed, or still registering; plugin ID mismatch between the snapshot/volume registration and the running plugin job.","commonSituations":"Controller plugin job stopped/failed before cleanup ran; node drains or client restarts leaving stale plugin registrations; plugin updated with a new ID; snapshot garbage collection racing plugin shutdown.","solutions":["Let Nomad retry — the retryable marker routes the RPC to another healthy controller","Restore/restart the controller plugin job (`nomad job start`/`nomad job revert`)","Verify with `nomad plugin status` that a controller plugin for the storage provider is healthy","Re-run the snapshot deletion once the plugin re-registers"],"exampleFix":"// before\nplugin_id = \"ebs-controller-v1\"  // job updated, plugin now registers as v2\n// after\nplugin_id = \"ebs-controller-v2\"","handlingStrategy":"retry","validationCode":"// Confirm a healthy controller exists before deleting snapshots:\n// nomad plugin status\n// and check the plugin job: nomad job status <csi-plugin-job>","typeGuard":null,"tryCatchPattern":"// Retryable: deletion will be re-routed to another controller\nif err := c.ControllerDeleteSnapshot(req, resp); err != nil {\n    if errors.Is(err, structs.ErrCSIClientRPCRetryable) {\n        // retry / wait for plugin re-registration\n    }\n}","preventionTips":["Keep controller plugin jobs highly available (restart stanza, max_parallel updates)","Don't stop plugin jobs before snapshot cleanup completes","Match plugin IDs across volume/snapshot registrations","Monitor plugin health to catch deregistrations early"],"tags":["csi","snapshot","plugin","retry"],"backgroundTag":"csi-plugin-not-found-retryable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}