{"record":{"id":"4019000195b4d71c","repo":"hashicorp/nomad","slug":"snapshot-q-has-a-pending-operation-v","errorCode":null,"errorMessage":"snapshot %q has a pending operation: %v","messagePattern":"snapshot %q has a pending operation: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"plugins/csi/client.go","lineNumber":698,"sourceCode":"\terr := req.Validate()\n\tif err != nil {\n\t\treturn err\n\t}\n\tcreq := req.ToCSIRepresentation()\n\t_, err = c.controllerClient.DeleteSnapshot(ctx, creq, opts...)\n\n\t// these standard gRPC error codes are overloaded with CSI-specific\n\t// meanings, so translate them into user-understandable terms\n\t// https://github.com/container-storage-interface/spec/blob/master/spec.md#deletesnapshot-errors\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.FailedPrecondition:\n\t\t\treturn fmt.Errorf(\n\t\t\t\t\"snapshot %q could not be deleted because it is in use: %v\",\n\t\t\t\treq.SnapshotID, err)\n\t\tcase codes.Aborted:\n\t\t\treturn fmt.Errorf(\"snapshot %q has a pending operation: %v\", req.SnapshotID, err)\n\t\tcase codes.Internal:\n\t\t\treturn fmt.Errorf(\n\t\t\t\t\"controller plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\t}\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\nfunc (c *client) ControllerListSnapshots(ctx context.Context, req *ControllerListSnapshotsRequest, opts ...grpc.CallOption) (*ControllerListSnapshotsResponse, error) {\n\tif err := c.ensureConnected(ctx); err != nil {\n\t\treturn nil, err\n\t}\n\n\terr := req.Validate()\n\tif err != nil {\n\t\treturn nil, err","sourceCodeStart":680,"sourceCodeEnd":716,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L680-L716","documentation":"In ControllerDeleteSnapshot, gRPC codes.Aborted per the CSI spec means the snapshot has another operation pending on it, so the delete was refused. Nomad renders 'snapshot %q has a pending operation' to indicate a transient conflict: the delete can succeed once the other operation completes.","triggerScenarios":"Calling ControllerDeleteSnapshot while a create/restore/expand operation involving req.SnapshotID is still in progress on the backend and the plugin returns codes.Aborted (client.go:698).","commonSituations":"Retention job deleting snapshots while a snapshot is mid-creation; concurrent delete requests from multiple Nomad clients; a restore from the snapshot still running; slow backend operations overlapping automated cleanup windows.","solutions":["Retry the delete after the pending operation completes, using backoff (not an immediate tight loop).","Serialize snapshot lifecycle operations per snapshot/volume to avoid overlap.","Skip snapshots marked in-flight in your scheduler and delete them in a later pass.","Check backend/plugin state to identify which operation is pending and wait for it to finish or fail."],"exampleFix":"// before: instant delete on schedule\nerr := deleteSnapshot(id)\n// after: retry with backoff when pending\nerr := retry.OnError(wait.Backoff{Steps: 5, Duration: 10 * time.Second},\n  func(e error) bool { return strings.Contains(e.Error(), \"pending operation\") },\n  func() error { return deleteSnapshot(id) })","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := retry.OnError(wait.Backoff{Steps: 5, Duration: 15 * time.Second, Factor: 2},\n  func(e error) bool { return strings.Contains(e.Error(), \"pending operation\") },\n  func() error { return c.ControllerDeleteSnapshot(ctx, req) })","preventionTips":["Serialize delete/create/restore operations per snapshot.","Schedule pruning outside snapshot/restore windows.","Skip in-flight snapshots and clean them up in a later pass."],"tags":["csi","snapshot","grpc","concurrency","nomad"],"backgroundTag":"operation-already-pending","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}