hashicorp/nomad · error

controller delete snapshot: %v

Error message

controller delete snapshot: %v

What it means

Nomad's ClientCSI.ControllerDeleteSnapshot forwards a CSI.ControllerDeleteSnapshot RPC to a controller plugin via sendCSIControllerRPC, wrapping any failure as 'controller delete snapshot: %v'. The snapshot deletion did not reach (or was rejected by) the storage backend.

Source

Thrown at nomad/client_csi_endpoint.go:154

		"ClientCSI.ControllerCreateSnapshot",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller create snapshot: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerDeleteSnapshot(args *cstructs.ClientCSIControllerDeleteSnapshotRequest, reply *cstructs.ClientCSIControllerDeleteSnapshotResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "delete_snapshot"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerDeleteSnapshot",
		"ClientCSI.ControllerDeleteSnapshot",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller delete snapshot: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerListSnapshots(args *cstructs.ClientCSIControllerListSnapshotsRequest, reply *cstructs.ClientCSIControllerListSnapshotsResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "list_snapshots"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerListSnapshots",
		"ClientCSI.ControllerListSnapshots",
		structs.RateMetricList,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller list snapshots: %v", err)
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify controller plugin health with `nomad plugin status <pluginID>` and restart its job if needed.
2. If the snapshot no longer exists in the backend, the delete may have effectively succeeded; force-deregister the snapshot record if Nomad tracks it.
3. Inspect server and plugin logs for the wrapped root error.
4. Retry after the plugin re-registers and reports healthy.

Example fix

# before: deleting snapshot with dead controller fails
nomad volume snapshot delete aws-ebs-controller snap-123
# after: restart controller, confirm healthy, then delete
nomad job run aws-ebs-csi-controller.nomad.hcl
nomad plugin status aws-ebs-controller
nomad volume snapshot delete aws-ebs-controller snap-123

// confirm the controller is up and the snapshot id is well-formed
if p.ControllersHealthy == 0 || !strings.HasPrefix(snapID, "snap-") {
    return fmt.Errorf("invalid delete-snapshot preconditions")
}

func controllerReady(p *api.CSIPlugin) bool { return p != nil && p.ControllersHealthy > 0 }

err := csi.ControllerDeleteSnapshot(args, reply)
if err != nil {
    // treat 'not found' style wrapped errors as idempotent success
    log.Printf("delete snapshot: %v", err)
}

When it happens

Trigger: Calling ControllerDeleteSnapshot (e.g. `nomad volume snapshot delete`) with no healthy controller instance, a stale plugin registration, or when the plugin/driver rejects the delete (snapshot already gone, backend error).

Common situations: Controller plugin deregistered or dead; snapshot was already deleted out-of-band causing driver errors; connectivity loss to the client hosting the plugin; driver-specific delete restrictions (clone in use).

Related errors

• controller create snapshot: %v
• controller list snapshots: %v
• volume snapshot ID cannot be updated
• controller list volumes: %v
• %s error: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/cff3a0eddc8d748b. Report an issue: GitHub.