{"record":{"id":"6671e87788605360","repo":"hashicorp/nomad","slug":"snapshot-q-is-already-pending-v","errorCode":null,"errorMessage":"snapshot %q is already pending: %v","messagePattern":"snapshot %q is already pending: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"plugins/csi/client.go","lineNumber":650,"sourceCode":"\terr := req.Validate()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tcreq := req.ToCSIRepresentation()\n\tresp, err := c.controllerClient.CreateSnapshot(ctx, creq, opts...)\n\n\t// these standard gRPC error codes are overloaded with CSI-specific\n\t// meanings, so translate them into user-understandable terms\n\t// https://github.com/container-storage-interface/spec/blob/master/spec.md#createsnapshot-errors\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.AlreadyExists:\n\t\t\treturn nil, fmt.Errorf(\n\t\t\t\t\"snapshot %q already exists but is incompatible with volume ID %q: %v\",\n\t\t\t\treq.Name, req.VolumeID, err)\n\t\tcase codes.Aborted:\n\t\t\treturn nil, fmt.Errorf(\n\t\t\t\t\"snapshot %q is already pending: %v\",\n\t\t\t\treq.Name, err)\n\t\tcase codes.ResourceExhausted:\n\t\t\treturn nil, fmt.Errorf(\n\t\t\t\t\"storage provider does not have enough space for this snapshot: %v\", err)\n\t\tcase codes.Internal:\n\t\t\treturn nil, fmt.Errorf(\n\t\t\t\t\"controller plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\t}\n\t\treturn nil, err\n\t}\n\n\tsnap := resp.GetSnapshot()\n\treturn &ControllerCreateSnapshotResponse{\n\t\tSnapshot: &Snapshot{\n\t\t\tID:             snap.GetSnapshotId(),\n\t\t\tSourceVolumeID: snap.GetSourceVolumeId(),\n\t\t\tSizeBytes:      snap.GetSizeBytes(),","sourceCodeStart":632,"sourceCodeEnd":668,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L632-L668","documentation":"In ControllerCreateSnapshot, the CSI spec maps gRPC codes.Aborted to 'a snapshot for this source volume is already in progress'. Nomad rewrites this as 'snapshot %q is already pending' so users know the creation is a duplicate in-flight operation, not a permanent failure. It is transient: retrying after the current snapshot completes should succeed.","triggerScenarios":"Calling ControllerCreateSnapshot while another CreateSnapshot for the same source volume (req.VolumeID) is still in flight on the storage backend and the plugin returns codes.Aborted (client.go:650).","commonSituations":"Overlapping periodic snapshot schedules; a retry fired while the first request was still being processed; slow backend (large volume) causing long-running snapshot that outlives a client timeout and triggers re-submission; concurrent Nomad jobs snapshotting the same volume.","solutions":["Wait for the in-flight snapshot to finish, then retry the CreateSnapshot call.","Serialize snapshot creation for the volume (single scheduler/locking) instead of issuing concurrent requests.","Increase client timeout/deadline so slow snapshots are not abandoned and retried while still running.","Check the plugin logs/backend UI to confirm the pending snapshot completed or failed before retrying."],"exampleFix":"// before: immediate retry loops on Aborted\nresp, err := c.ControllerCreateSnapshot(ctx, req)\n// after: backoff on Aborted (pending)\nresp, err := c.ControllerCreateSnapshot(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"already pending\") {\n  time.Sleep(30 * time.Second)\n  resp, err = c.ControllerCreateSnapshot(ctx, req)\n}","handlingStrategy":"retry","validationCode":"func snapshotPending(ctx context.Context, c *client, volumeID string) (bool, error) {\n  // track in-flight creates in-process or via backend listing\n  resp, err := c.ControllerListSnapshots(ctx, &ControllerListSnapshotsRequest{})\n  if err != nil { return false, err }\n  for _, s := range resp.Snapshots {\n    if s.SourceVolumeID == volumeID && !s.IsReady { return true, nil }\n  }\n  return false, nil\n}","typeGuard":null,"tryCatchPattern":"err := retry.OnError(wait.Backoff{Steps: 6, Duration: 30 * time.Second, Factor: 2},\n  func(e error) bool { return strings.Contains(e.Error(), \"already pending\") },\n  func() error { _, e := c.ControllerCreateSnapshot(ctx, req); return e })","preventionTips":["Serialize snapshot creation per volume with a lock/single scheduler.","Set RPC deadlines longer than the backend's worst-case snapshot time.","Avoid overlapping periodic snapshot jobs on the same volume."],"tags":["csi","snapshot","grpc","concurrency","nomad"],"backgroundTag":"operation-already-pending","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}