hashicorp/nomad · warning

snapshot %q is already pending: %v

Error message

snapshot %q is already pending: %v

What it means

In ControllerCreateSnapshot, the CSI spec maps gRPC codes.Aborted to 'a snapshot for this source volume is already in progress'. Nomad rewrites this as 'snapshot %q is already pending' so users know the creation is a duplicate in-flight operation, not a permanent failure. It is transient: retrying after the current snapshot completes should succeed.

Source

Thrown at plugins/csi/client.go:650

	err := req.Validate()
	if err != nil {
		return nil, err
	}
	creq := req.ToCSIRepresentation()
	resp, err := c.controllerClient.CreateSnapshot(ctx, creq, opts...)

	// these standard gRPC error codes are overloaded with CSI-specific
	// meanings, so translate them into user-understandable terms
	// https://github.com/container-storage-interface/spec/blob/master/spec.md#createsnapshot-errors
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.AlreadyExists:
			return nil, fmt.Errorf(
				"snapshot %q already exists but is incompatible with volume ID %q: %v",
				req.Name, req.VolumeID, err)
		case codes.Aborted:
			return nil, fmt.Errorf(
				"snapshot %q is already pending: %v",
				req.Name, err)
		case codes.ResourceExhausted:
			return nil, fmt.Errorf(
				"storage provider does not have enough space for this snapshot: %v", err)
		case codes.Internal:
			return nil, fmt.Errorf(
				"controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return nil, err
	}

	snap := resp.GetSnapshot()
	return &ControllerCreateSnapshotResponse{
		Snapshot: &Snapshot{
			ID:             snap.GetSnapshotId(),
			SourceVolumeID: snap.GetSourceVolumeId(),
			SizeBytes:      snap.GetSizeBytes(),

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Wait for the in-flight snapshot to finish, then retry the CreateSnapshot call.
2. Serialize snapshot creation for the volume (single scheduler/locking) instead of issuing concurrent requests.
3. Increase client timeout/deadline so slow snapshots are not abandoned and retried while still running.
4. Check the plugin logs/backend UI to confirm the pending snapshot completed or failed before retrying.

Example fix

// before: immediate retry loops on Aborted
resp, err := c.ControllerCreateSnapshot(ctx, req)
// after: backoff on Aborted (pending)
resp, err := c.ControllerCreateSnapshot(ctx, req)
if err != nil && strings.Contains(err.Error(), "already pending") {
  time.Sleep(30 * time.Second)
  resp, err = c.ControllerCreateSnapshot(ctx, req)
}

func snapshotPending(ctx context.Context, c *client, volumeID string) (bool, error) {
  // track in-flight creates in-process or via backend listing
  resp, err := c.ControllerListSnapshots(ctx, &ControllerListSnapshotsRequest{})
  if err != nil { return false, err }
  for _, s := range resp.Snapshots {
    if s.SourceVolumeID == volumeID && !s.IsReady { return true, nil }
  }
  return false, nil
}

err := retry.OnError(wait.Backoff{Steps: 6, Duration: 30 * time.Second, Factor: 2},
  func(e error) bool { return strings.Contains(e.Error(), "already pending") },
  func() error { _, e := c.ControllerCreateSnapshot(ctx, req); return e })

When it happens

Trigger: Calling ControllerCreateSnapshot while another CreateSnapshot for the same source volume (req.VolumeID) is still in flight on the storage backend and the plugin returns codes.Aborted (client.go:650).

Common situations: Overlapping periodic snapshot schedules; a retry fired while the first request was still being processed; slow backend (large volume) causing long-running snapshot that outlives a client timeout and triggers re-submission; concurrent Nomad jobs snapshotting the same volume.

Related errors

• snapshot %q has a pending operation: %v
• snapshot %q already exists but is incompatible with volume I
• storage provider does not have enough space for this snapsho
• snapshot %q could not be deleted because it is in use: %v
• volume snapshot ID cannot be updated

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6671e87788605360. Report an issue: GitHub.