hashicorp/nomad · error

controller create snapshot: %v

Error message

controller create snapshot: %v

What it means

Nomad's ClientCSI.ControllerCreateSnapshot forwards a CSI.ControllerCreateSnapshot RPC to a healthy controller plugin instance via sendCSIControllerRPC. On any forwarding or plugin RPC failure the error is wrapped as 'controller create snapshot: %v'. The snapshot was not created.

Source

Thrown at nomad/client_csi_endpoint.go:140

		"ClientCSI.ControllerListVolumes",
		structs.RateMetricList,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller list volumes: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerCreateSnapshot(args *cstructs.ClientCSIControllerCreateSnapshotRequest, reply *cstructs.ClientCSIControllerCreateSnapshotResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "create_snapshot"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerCreateSnapshot",
		"ClientCSI.ControllerCreateSnapshot",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller create snapshot: %v", err)
	}
	return nil
}

func (a *ClientCSI) ControllerDeleteSnapshot(args *cstructs.ClientCSIControllerDeleteSnapshotRequest, reply *cstructs.ClientCSIControllerDeleteSnapshotResponse) error {
	defer metrics.MeasureSince([]string{"nomad", "client_csi_controller", "delete_snapshot"}, time.Now())

	err := a.sendCSIControllerRPC(args.PluginID,
		"CSI.ControllerDeleteSnapshot",
		"ClientCSI.ControllerDeleteSnapshot",
		structs.RateMetricWrite,
		args, reply)
	if err != nil {
		return fmt.Errorf("controller delete snapshot: %v", err)
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Run `nomad plugin status <pluginID>` and confirm a healthy controller instance exists.
2. Ensure the controller plugin job is running and connected (`nomad node status` shows the client).
3. Check the CSI driver logs for backend-side snapshot failures (permissions, quotas, driver limits).
4. Retry the snapshot command once connectivity/health is restored.

Example fix

# before: controller job stopped
nomad job stop aws-ebs-csi-controller
# after: redeploy and verify health before snapshotting
nomad job run aws-ebs-csi-controller.nomad.hcl
nomad plugin status aws-ebs-controller
nomad volume snapshot create ebs-vol my-snapshot

// verify plugin health and volume existence first
p, _, _ := client.Plugins().Info(ctx, pluginID, nil)
v, _, _ := client.Volumes().Info(ctx, volID, nil)
if p.ControllersHealthy == 0 || v == nil {
    return fmt.Errorf("cannot snapshot: plugin or volume not ready")
}

func canSnapshot(p *api.CSIPlugin, v *api.CSIVolume) bool {
    return p != nil && p.ControllersHealthy > 0 && v != nil
}

err := csi.ControllerCreateSnapshot(args, reply)
if err != nil {
    if strings.Contains(err.Error(), "controller create snapshot:") {
        // check plugin logs, retry after health restored
    }
    return err
}

When it happens

Trigger: Calling ControllerCreateSnapshot (e.g. `nomad volume snapshot create`) when the controller plugin is missing, unhealthy, unreachable, or the CSI plugin rejects the snapshot request (unsupported volume, driver error).

Common situations: Controller plugin job not running; volume's pluginID mismatched; storage backend rejecting snapshot (volume in use, quota exceeded); server-to-client RPC timeout.

Related errors

• controller delete snapshot: %v
• controller list snapshots: %v
• volume snapshot ID cannot be updated
• controller list volumes: %v
• %s error: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fa3bba8ced2e3053. Report an issue: GitHub.