{"record":{"id":"312bbb33422bdc7e","repo":"hashicorp/nomad","slug":"could-not-delete-q-v","errorCode":null,"errorMessage":"could not delete %q: %v","messagePattern":"could not delete %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/csi_endpoint.go","lineNumber":1743,"sourceCode":"\t\t}\n\t\tif !plugin.HasControllerCapability(structs.CSIControllerSupportsCreateDeleteSnapshot) {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"plugin does not support snapshot\"))\n\t\t\tcontinue\n\t\t}\n\n\t\tmethod := \"ClientCSI.ControllerDeleteSnapshot\"\n\n\t\tcReq := &cstructs.ClientCSIControllerDeleteSnapshotRequest{\n\t\t\tID:      snap.ID,\n\t\t\tSecrets: snap.Secrets,\n\t\t}\n\t\tcReq.PluginID = plugin.ID\n\t\tcResp := &cstructs.ClientCSIControllerDeleteSnapshotResponse{}\n\t\terr = v.serializedControllerRPC(plugin.ID, func() error {\n\t\t\treturn v.srv.RPC(method, cReq, cResp)\n\t\t})\n\t\tif err != nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"could not delete %q: %v\", snap.ID, err))\n\t\t}\n\t}\n\treturn mErr.ErrorOrNil()\n}\n\nfunc (v *CSIVolume) ListSnapshots(args *structs.CSISnapshotListRequest, reply *structs.CSISnapshotListResponse) error {\n\n\tauthErr := v.srv.Authenticate(v.ctx, args)\n\tif done, err := v.srv.forward(\"CSIVolume.ListSnapshots\", args, args, reply); done {\n\t\treturn err\n\t}\n\tv.srv.MeasureRPCRate(\"csi_volume\", structs.RateMetricList, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"volume\", \"list_snapshots\"}, time.Now())\n\n\tallowVolume := acl.NamespaceValidator(acl.NamespaceCapabilityCSIListVolume,","sourceCodeStart":1725,"sourceCodeEnd":1761,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L1725-L1761","documentation":"Raised in CSIVolume.DeleteSnapshot when the serialized controller RPC for ClientCSI.ControllerDeleteSnapshot fails. The plugin was found and supports snapshots, but forwarding the delete to the client running the controller failed — the underlying error is wrapped as 'could not delete %q: %v' with the snapshot ID.","triggerScenarios":"Calling DeleteSnapshot where the controller RPC fails because the plugin's node/client is down, the controller is busy (serialized RPC timeout), the snapshot does not exist at the storage backend, secrets are wrong, or the CSI driver returned an error.","commonSituations":"Client running the controller is drained/down; snapshot already deleted externally (NotFound from provider); incorrect snapshot Secrets; controller RPC serialization contention on a heavily used plugin; CSI driver errors (e.g. EBS permissions).","solutions":["Read the wrapped inner error to distinguish NotFound (snapshot already gone — safe to ignore) from connectivity/permission failures.","Ensure the client node running the controller plugin is healthy and connected (`nomad node status`).","Verify snapshot Secrets are correct and sufficient for the storage provider operation.","Retry the delete; the controller RPC is serialized per plugin so transient contention resolves with retry.","Check the CSI driver/provider logs for the underlying delete error (e.g. cloud IAM permissions)."],"exampleFix":"// before — deleting with missing secrets\nsnaps := []*structs.CSISnapshot{{ID: id}}\nclient.CSI().DeleteSnapshot(&structs.CSISnapshotDeleteRequest{Snapshots: snaps})\n// after — provide required provider secrets\nsnaps := []*structs.CSISnapshot{{ID: id, Secrets: structs.CSISecrets{\"region\": \"us-east-1\"}}}\nclient.CSI().DeleteSnapshot(&structs.CSISnapshotDeleteRequest{Snapshots: snaps})","handlingStrategy":"retry","validationCode":"// ensure the controller's node is healthy and the snapshot exists before deleting\nnodes, _, _ := client.Nodes().List(nil)\nhealthy := false\nfor _, n := range nodes {\n    if ni, _, _ := client.Nodes().Info(n.ID, nil); ni != nil && ni.Status == \"ready\" && ni.Drain == false { healthy = true; break }\n}\nif !healthy { return errors.New(\"no healthy node for controller RPC\") }","typeGuard":null,"tryCatchPattern":"var mErr *multijson.Error // DeleteSnapshot returns multierror\nif err := client.CSI().DeleteSnapshot(req); err != nil {\n    for _, sub := range multijson.Errors(err) {\n        if strings.Contains(sub.Error(), \"could not delete\") {\n            if strings.Contains(sub.Error(), \"NotFound\") {\n                continue // already deleted at provider\n            }\n            // retry once for transient controller failures\n            _ = client.CSI().DeleteSnapshot(req)\n        }\n    }\n}","preventionTips":["Keep the client node running the controller healthy and undrained.","Provide correct snapshot Secrets for the storage provider.","Retry deletes — the controller RPC is serialized per plugin and can time out under contention.","Treat NotFound in the wrapped error as success (idempotent delete).","Check CSI driver and cloud provider logs for root-cause errors."],"tags":["csi","rpc","snapshot","controller"],"backgroundTag":"csi-controller-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}