hashicorp/nomad · error

could not delete %q: %v

Error message

could not delete %q: %v

What it means

Raised in CSIVolume.DeleteSnapshot when the serialized controller RPC for ClientCSI.ControllerDeleteSnapshot fails. The plugin was found and supports snapshots, but forwarding the delete to the client running the controller failed — the underlying error is wrapped as 'could not delete %q: %v' with the snapshot ID.

Source

Thrown at nomad/csi_endpoint.go:1743

		}
		if !plugin.HasControllerCapability(structs.CSIControllerSupportsCreateDeleteSnapshot) {
			multierror.Append(&mErr, fmt.Errorf("plugin does not support snapshot"))
			continue
		}

		method := "ClientCSI.ControllerDeleteSnapshot"

		cReq := &cstructs.ClientCSIControllerDeleteSnapshotRequest{
			ID:      snap.ID,
			Secrets: snap.Secrets,
		}
		cReq.PluginID = plugin.ID
		cResp := &cstructs.ClientCSIControllerDeleteSnapshotResponse{}
		err = v.serializedControllerRPC(plugin.ID, func() error {
			return v.srv.RPC(method, cReq, cResp)
		})
		if err != nil {
			multierror.Append(&mErr, fmt.Errorf("could not delete %q: %v", snap.ID, err))
		}
	}
	return mErr.ErrorOrNil()
}

func (v *CSIVolume) ListSnapshots(args *structs.CSISnapshotListRequest, reply *structs.CSISnapshotListResponse) error {

	authErr := v.srv.Authenticate(v.ctx, args)
	if done, err := v.srv.forward("CSIVolume.ListSnapshots", args, args, reply); done {
		return err
	}
	v.srv.MeasureRPCRate("csi_volume", structs.RateMetricList, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "volume", "list_snapshots"}, time.Now())

	allowVolume := acl.NamespaceValidator(acl.NamespaceCapabilityCSIListVolume,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped inner error to distinguish NotFound (snapshot already gone — safe to ignore) from connectivity/permission failures.
2. Ensure the client node running the controller plugin is healthy and connected (`nomad node status`).
3. Verify snapshot Secrets are correct and sufficient for the storage provider operation.
4. Retry the delete; the controller RPC is serialized per plugin so transient contention resolves with retry.
5. Check the CSI driver/provider logs for the underlying delete error (e.g. cloud IAM permissions).

Example fix

// before — deleting with missing secrets
snaps := []*structs.CSISnapshot{{ID: id}}
client.CSI().DeleteSnapshot(&structs.CSISnapshotDeleteRequest{Snapshots: snaps})
// after — provide required provider secrets
snaps := []*structs.CSISnapshot{{ID: id, Secrets: structs.CSISecrets{"region": "us-east-1"}}}
client.CSI().DeleteSnapshot(&structs.CSISnapshotDeleteRequest{Snapshots: snaps})

// ensure the controller's node is healthy and the snapshot exists before deleting
nodes, _, _ := client.Nodes().List(nil)
healthy := false
for _, n := range nodes {
    if ni, _, _ := client.Nodes().Info(n.ID, nil); ni != nil && ni.Status == "ready" && ni.Drain == false { healthy = true; break }
}
if !healthy { return errors.New("no healthy node for controller RPC") }

var mErr *multijson.Error // DeleteSnapshot returns multierror
if err := client.CSI().DeleteSnapshot(req); err != nil {
    for _, sub := range multijson.Errors(err) {
        if strings.Contains(sub.Error(), "could not delete") {
            if strings.Contains(sub.Error(), "NotFound") {
                continue // already deleted at provider
            }
            // retry once for transient controller failures
            _ = client.CSI().DeleteSnapshot(req)
        }
    }
}

When it happens

Trigger: Calling DeleteSnapshot where the controller RPC fails because the plugin's node/client is down, the controller is busy (serialized RPC timeout), the snapshot does not exist at the storage backend, secrets are wrong, or the CSI driver returned an error.

Common situations: Client running the controller is drained/down; snapshot already deleted externally (NotFound from provider); incorrect snapshot Secrets; controller RPC serialization contention on a heavily used plugin; CSI driver errors (e.g. EBS permissions).

Related errors

• could not create snapshot: %v
• plugin does not support snapshot
• cannot upload management tokens
• volume snapshot ID cannot be updated
• missing VolumeID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/312bbb33422bdc7e. Report an issue: GitHub.