hashicorp/nomad · error

CSI.ControllerDeleteVolume: %w: %v

Error message

CSI.ControllerDeleteVolume: %w: %v

What it means

Nomad's ControllerDeleteVolume endpoint could not find a healthy controller plugin instance matching req.PluginID (findControllerPlugin failed). The error is wrapped with structs.ErrCSIClientRPCRetryable so the Nomad server knows the plugin health view may be stale and will retry the RPC, possibly against another controller instance.

Source

Thrown at client/csi_endpoint.go:284

		return fmt.Errorf("CSI.ControllerExpandVolume: %v", err)
	}
	if cresp == nil {
		c.c.logger.Warn("plugin did not return error or response; this is a bug in the plugin and should be reported to the plugin author")
		return fmt.Errorf("CSI.ControllerExpandVolume: plugin did not return error or response")
	}
	resp.CapacityBytes = cresp.CapacityBytes
	resp.NodeExpansionRequired = cresp.NodeExpansionRequired
	return nil
}

func (c *CSI) ControllerDeleteVolume(req *structs.ClientCSIControllerDeleteVolumeRequest, resp *structs.ClientCSIControllerDeleteVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "delete_volume"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerDeleteVolume: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerDeleteVolume errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	err = plugin.ControllerDeleteVolume(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the volume was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the delete — the error is explicitly retryable and Nomad will reschedule it once a healthy controller is available
2. Verify the CSI plugin controller task is running and healthy (nomad job status <plugin-job>; nomad node status -verbose)
3. Confirm the plugin_id in the volume spec matches the controller plugin's ID (nomad plugin status <id>)
4. If the plugin is gone permanently, re-register the plugin job or deregister the stale volume

Example fix

// before: volume spec pointing at stale plugin
plugin_id = "aws-ebs-controller-old"
// after
plugin_id = "aws-ebs-controller"

plug, err := apiClient.Plugins().Get("aws-ebs-controller")
if err != nil {
    return fmt.Errorf("plugin not registered: %w", err)
}
if plug.ControllersHealthy < 1 {
    return fmt.Errorf("no healthy controller instances for %s", plug.ID)
}

func isCSIRetryable(err error) bool {
    return err != nil && errors.Is(err, structs.ErrCSIClientRPCRetryable)
}

err := client.CSI().ControllerDeleteVolume(req)
if errors.Is(err, structs.ErrCSIClientRPCRetryable) {
    // wait for the server to re-route to another controller instance
    time.Sleep(2 * time.Second)
    return retryDelete(req)
}
return err

When it happens

Trigger: Calling volume deletion (nomad volume delete / deregister flow) while the plugin's controller is not running, has failed its health check, has just been restarted, or the plugin ID in the request does not match any controller on the client.

Common situations: CSI plugin task crashed or is still starting up when a job/volume cleanup runs; node was drained or the plugin was upgraded; typo'd or stale plugin ID in the volume specification; Nomad server's catalog is out of sync after client restart.

Related errors

• CSI.ControllerListVolumes: %w: %v
• CSI.ControllerExpandVolume could not find plugin: %w: %v (wr
• %v: %v
• no such plugin
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c4b92fc180d64029. Report an issue: GitHub.