hashicorp/nomad · error · ErrCSIClientRPCRetryable

CSI.ControllerCreateVolume: %w: %v (wraps ErrCSIClientRPCRet

Error message

CSI.ControllerCreateVolume: %w: %v (wraps ErrCSIClientRPCRetryable)

What it means

ControllerCreateVolume wraps a findControllerPlugin failure in ErrCSIClientRPCRetryable, telling the server its plugin-health view is stale and it should retry with another controller instance. The underlying err explains why no controller plugin matched req.PluginID.

Source

Thrown at client/csi_endpoint.go:192

		// if the controller detach previously happened but the server failed to
		// checkpoint, we'll get an error from the plugin but can safely ignore it.
		c.c.logger.Debug("could not unpublish volume", "error", err)
		return nil
	}
	if err != nil {
		return fmt.Errorf("CSI.ControllerDetachVolume: %v", err)
	}
	return err
}

func (c *CSI) ControllerCreateVolume(req *structs.ClientCSIControllerCreateVolumeRequest, resp *structs.ClientCSIControllerCreateVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "create_volume"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerCreateVolume: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq, err := req.ToCSIRequest()
	if err != nil {
		return fmt.Errorf("CSI.ControllerCreateVolume: %v", err)
	}

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerCreateVolume errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	cresp, err := plugin.ControllerCreateVolume(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Ensure the controller plugin job is running and healthy (`nomad plugin status <plugin>`), then retry volume creation — the error is deliberately retryable
  2. Check the plugin is controller-capable (type controller) and the plugin ID in the volume spec matches
  3. Restart or reschedule the plugin allocation if it is unhealthy
  4. If the controller moved, allow the server to re-route to the healthy controller instance

Example fix

// before
volume {
  type = "csi"
  plugin_id = "ebs"   # job registers plugin as "ebs-csi"
}
// after
volume {
  type = "csi"
  plugin_id = "ebs-csi"
}
Defensive patterns

Strategy: retry

Validate before calling

// before volume create, confirm a healthy controller plugin is registered
p := serverCSIPlugin(pluginID)
if p == nil || p.ControllerRequired() == false || p.Healthy() != nil {
    return fmt.Errorf("controller plugin %s not available for volume creation", pluginID)
}

Type guard

func isRetryableCSI(err error) bool { return structs.IsErrRetryable(err) }

Try / catch

err := c.ControllerCreateVolume(req, resp)
if err != nil {
    if structs.IsErrRetryable(err) {
        // wait for plugin health to refresh, then retry with backoff
        time.Sleep(retryDelay)
        return c.ControllerCreateVolume(req, resp)
    }
    return err
}

Prevention

When it happens

Trigger: A ClientCSIControllerCreateVolume RPC (from `nomad volume create` or external volume registration) reaches a client where no healthy controller plugin exists for req.PluginID.

Common situations: Dynamic volume provisioning while the controller plugin job is starting/crashed; plugin registered node-only so it cannot create volumes; cluster scale-down removed the node hosting the controller; plugin ID mismatch after job rename.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/13ac3717427ee109. Report an issue: GitHub.