hashicorp/nomad · warning

snapshot %q has a pending operation: %v

Error message

snapshot %q has a pending operation: %v

What it means

In ControllerDeleteSnapshot, gRPC codes.Aborted per the CSI spec means the snapshot has another operation pending on it, so the delete was refused. Nomad renders 'snapshot %q has a pending operation' to indicate a transient conflict: the delete can succeed once the other operation completes.

Source

Thrown at plugins/csi/client.go:698

	err := req.Validate()
	if err != nil {
		return err
	}
	creq := req.ToCSIRepresentation()
	_, err = c.controllerClient.DeleteSnapshot(ctx, creq, opts...)

	// these standard gRPC error codes are overloaded with CSI-specific
	// meanings, so translate them into user-understandable terms
	// https://github.com/container-storage-interface/spec/blob/master/spec.md#deletesnapshot-errors
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.FailedPrecondition:
			return fmt.Errorf(
				"snapshot %q could not be deleted because it is in use: %v",
				req.SnapshotID, err)
		case codes.Aborted:
			return fmt.Errorf("snapshot %q has a pending operation: %v", req.SnapshotID, err)
		case codes.Internal:
			return fmt.Errorf(
				"controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return err
	}

	return nil
}

func (c *client) ControllerListSnapshots(ctx context.Context, req *ControllerListSnapshotsRequest, opts ...grpc.CallOption) (*ControllerListSnapshotsResponse, error) {
	if err := c.ensureConnected(ctx); err != nil {
		return nil, err
	}

	err := req.Validate()
	if err != nil {
		return nil, err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the delete after the pending operation completes, using backoff (not an immediate tight loop).
2. Serialize snapshot lifecycle operations per snapshot/volume to avoid overlap.
3. Skip snapshots marked in-flight in your scheduler and delete them in a later pass.
4. Check backend/plugin state to identify which operation is pending and wait for it to finish or fail.

Example fix

// before: instant delete on schedule
err := deleteSnapshot(id)
// after: retry with backoff when pending
err := retry.OnError(wait.Backoff{Steps: 5, Duration: 10 * time.Second},
  func(e error) bool { return strings.Contains(e.Error(), "pending operation") },
  func() error { return deleteSnapshot(id) })

err := retry.OnError(wait.Backoff{Steps: 5, Duration: 15 * time.Second, Factor: 2},
  func(e error) bool { return strings.Contains(e.Error(), "pending operation") },
  func() error { return c.ControllerDeleteSnapshot(ctx, req) })

When it happens

Trigger: Calling ControllerDeleteSnapshot while a create/restore/expand operation involving req.SnapshotID is still in progress on the backend and the plugin returns codes.Aborted (client.go:698).

Common situations: Retention job deleting snapshots while a snapshot is mid-creation; concurrent delete requests from multiple Nomad clients; a restore from the snapshot still running; slow backend operations overlapping automated cleanup windows.

Related errors

• snapshot %q is already pending: %v
• snapshot %q already exists but is incompatible with volume I
• storage provider does not have enough space for this snapsho
• snapshot %q could not be deleted because it is in use: %v
• volume snapshot ID cannot be updated

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4019000195b4d71c. Report an issue: GitHub.