hashicorp/nomad · error

CSI.ControllerDetachVolume: %v

Error message

CSI.ControllerDetachVolume: %v

What it means

The plugin's ControllerUnpublishVolume gRPC call failed and the failure was not the ignorable 'already detached' case (which is logged at debug and swallowed). The error surfaces after up to 3 retries, prefixed with CSI.ControllerDetachVolume; it means the node's unpublish checkpoint could not be completed via the controller.

Source

Thrown at client/csi_endpoint.go:180

	csiReq := req.ToCSIRequest()

	// Submit the request for a volume to the CSI Plugin.
	ctx, cancelFn := c.requestContext()
	defer cancelFn()
	// CSI ControllerUnpublishVolume errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	_, err = plugin.ControllerUnpublishVolume(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the controller detach previously happened but the server failed to
		// checkpoint, we'll get an error from the plugin but can safely ignore it.
		c.c.logger.Debug("could not unpublish volume", "error", err)
		return nil
	}
	if err != nil {
		return fmt.Errorf("CSI.ControllerDetachVolume: %v", err)
	}
	return err
}

func (c *CSI) ControllerCreateVolume(req *structs.ClientCSIControllerCreateVolumeRequest, resp *structs.ClientCSIControllerCreateVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "create_volume"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerCreateVolume: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq, err := req.ToCSIRequest()
	if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error: if the provider says volume not attached, the state is effectively detached — run `nomad volume detach` or re-run unpublish to reconcile the checkpoint
2. Check the plugin logs (`nomad alloc logs <plugin-alloc>`) for the underlying gRPC/provider error
3. Fix provider-side blockers (credentials, region, attachment state) and retry unpublish
4. As a last resort force-detach at the provider, then let Nomad GC/reconcile the volume claim

Example fix

// before
# unpublish keeps failing: attachment stale at provider
// after
aws ec2 detach-volume --volume-id vol-xxx --instance-id i-yyy --force
nomad volume detach <vol_id> <node_id>  # reconcile checkpoint

// reconcile first: if the provider shows no attachment, skip controller unpublish
if !providerHasAttachment(volumeExternalID, nodeID) {
    return nil // already detached; only checkpoint cleanup needed
}

err := c.ControllerDetachVolume(req, resp)
switch {
case err == nil:
    // detached
case isAlreadyDetached(err):
    // treat as success, reconcile claim state
case structs.IsErrRetryable(err):
    // retry with another controller
default:
    // inspect plugin logs, fix provider state, force-detach if needed
}

When it happens

Trigger: ControllerUnpublishVolume returned a non-retryable gRPC error from the plugin/provider (volume not found, permission error, provider API failure), or transient errors persisted through 3 retries.

Common situations: Provider reports the volume is not attached / wrong attachment state, so unpublish fails; expired provider credentials; plugin container unable to reach the storage API; double-detach races leaving inconsistent checkpoint state.

Related errors

• nil response from plugin.NodeExpandVolume
• CSI plugin failed to register: %w
• failed to probe plugin: %w
• CSI.ControllerDetachVolume: %w: %v (wraps ErrCSIClientRPCRet
• CSI.ControllerCreateVolume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a41b6088f14d619e. Report an issue: GitHub.