{"record":{"id":"eaae17c4c08d887b","repo":"hashicorp/nomad","slug":"csi-controllerdeletesnapshot-v","errorCode":null,"errorMessage":"CSI.ControllerDeleteSnapshot: %v","messagePattern":"CSI\\.ControllerDeleteSnapshot: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":442,"sourceCode":"\tcsiReq := req.ToCSIRequest()\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerDeleteSnapshot errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\terr = plugin.ControllerDeleteSnapshot(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {\n\t\t// if the snapshot was deleted out-of-band, we'll get an error from\n\t\t// the plugin but can safely ignore it\n\t\tc.c.logger.Debug(\"could not delete snapshot\", \"error\", err)\n\t\treturn nil\n\t}\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerDeleteSnapshot: %v\", err)\n\t}\n\treturn err\n}\n\nfunc (c *CSI) ControllerListSnapshots(req *structs.ClientCSIControllerListSnapshotsRequest, resp *structs.ClientCSIControllerListSnapshotsResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"list_snapshots\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerListSnapshots: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq := req.ToCSIRequest()\n","sourceCodeStart":424,"sourceCodeEnd":460,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L424-L460","documentation":"CSI.ControllerDeleteSnapshot wraps any non-ignored error from the plugin's ControllerDeleteSnapshot gRPC call with this message. Note the preceding branch: if the plugin reports the snapshot was already deleted out-of-band, Nomad logs at debug and returns nil. So this error means a genuine deletion failure that is not 'snapshot already gone'.","triggerScenarios":"plugin.ControllerDeleteSnapshot returns an error that is not the 'not found'/'deleted out-of-band' case: invalid snapshot ID, permissions failure, backend outage, or fatal CSI status code.","commonSituations":"Backend removed the snapshot and the plugin surfaces a non-standard error code Nomad doesn't recognize as not-found; IAM/credentials revoked between create and delete; storage backend API outage; deleting snapshots while the plugin is being upgraded.","solutions":["Inspect the wrapped gRPC/CSI status in the message to see the backend's actual complaint","Verify the snapshot ID exists in the storage backend; if already gone, retry — Nomad ignores not-found","Check plugin credentials/permissions to delete snapshots in the backend account","Check plugin logs and backend status, then retry the delete"],"exampleFix":"// before (plugin side): returning generic error when snapshot missing\nreturn nil, status.Error(codes.Internal, \"snapshot missing\")\n// after\nreturn nil, status.Error(codes.NotFound, \"snapshot already deleted\")","handlingStrategy":"try-catch","validationCode":"// Check the snapshot still exists before deleting:\n// nomad volume snapshot list -plugin <plugin-id>\n// Only issue DeleteSnapshot for snapshots present in the listing.","typeGuard":null,"tryCatchPattern":"// Distinguish not-found (ignored by Nomad) from real failures\nerr := c.ControllerDeleteSnapshot(req, resp)\nif err != nil {\n    if strings.Contains(err.Error(), \"CSI.ControllerDeleteSnapshot:\") {\n        // inspect wrapped gRPC/CSI status; treat NotFound as already-deleted success\n    }\n}","preventionTips":["Verify snapshot existence in the backend before deletion","Keep plugin IAM credentials valid for delete operations","Align plugin 'not found' semantics with CSI codes.NotFound so Nomad can ignore them","Retry idempotently — deletes should tolerate already-deleted snapshots"],"tags":["csi","grpc","snapshot","delete"],"backgroundTag":"grpc-internal-error","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}