hashicorp/nomad · error

controller plugin returned an error: %v

Error message

controller plugin returned an error: %v

What it means

This error wraps any gRPC status error returned by the CSI controller plugin during ControllerExpandVolume that is not OutOfRange or Internal (e.g. Unavailable, DeadlineExceeded, NotFound, Unimplemented). Nomad converts the plugin's gRPC error into a generic message prefixed with 'controller plugin returned an error' so the caller knows the failure originated in the external storage plugin, not Nomad itself.

Source

Thrown at plugins/csi/client.go:544

	resp, err := c.controllerClient.ControllerExpandVolume(ctx, exReq, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.InvalidArgument:
			return nil, fmt.Errorf(
				"requested capabilities not compatible with volume %q: %v",
				req.ExternalVolumeID, err)
		case codes.NotFound:
			err = fmt.Errorf("volume %q could not be found: %v", req.ExternalVolumeID, err)
		case codes.FailedPrecondition:
			err = fmt.Errorf("volume %q cannot be expanded online: %v", req.ExternalVolumeID, err)
		case codes.OutOfRange:
			return nil, fmt.Errorf(
				"unsupported capacity_range for volume %q: %v", req.ExternalVolumeID, err)
		case codes.Internal:
			err = fmt.Errorf("controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		default:
			err = fmt.Errorf("controller plugin returned an error: %v", err)
		}
		return nil, err
	}

	return &ControllerExpandVolumeResponse{
		CapacityBytes:         resp.GetCapacityBytes(),
		NodeExpansionRequired: resp.GetNodeExpansionRequired(),
	}, nil
}

// compareCapabilities returns an error if the 'got' capabilities aren't found
// within the 'expected' capability.
//
// Note that plugins in the wild are known to return incomplete
// VolumeCapability responses, so we can't require that all capabilities we
// expect have been validated, only that the ones that have been validated
// match. This appears to violate the CSI specification but until that's been
// resolved in upstream we have to loosen our validation requirements. The

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the %v suffix of the error for the underlying gRPC status and check the plugin's own logs for the root cause
2. Verify the ExternalVolumeID exists in the storage backend and matches what the plugin knows
3. Check plugin container health/restart it if the status is Unavailable or DeadlineExceeded
4. Confirm the plugin implements the CSI 1.x ControllerExpandVolume RPC (not a 'controller capabilities' limited plugin)
5. Retry the expansion if the status is transient (Unavailable/DeadlineExceeded)

Example fix

// before: expanding a volume whose ID no longer exists in the backend
client.ControllerExpandVolume(ctx, &csi.ControllerExpandVolumeRequest{ExternalVolumeID: "vol-stale-id", CapacityRange: ...})
// error: controller plugin returned an error: rpc error: code = NotFound ...

// after: verify the volume ID against the backend first, or list volumes via ControllerGetCapabilities/ListVolumes before expanding

// before expanding, confirm plugin is healthy and volume exists
resp, err := plugin.ControllerGetCapabilities(ctx, &csi.ControllerGetCapabilitiesRequest{})
if err != nil { return fmt.Errorf("plugin controller unavailable: %w", err) }
var hasExpand bool
for _, c := range resp.GetCapabilities() {
    if c.GetRpc().GetType() == csipbv1.ControllerServiceCapability_RPC_EXPAND_VOLUME { hasExpand = true }
}
if !hasExpand { return errors.New("plugin does not support ControllerExpandVolume") }

func isRetryableGRPCError(err error) bool {
    st, ok := status.FromError(err)
    if !ok { return false }
    switch st.Code() {
    case codes.Unavailable, codes.DeadlineExceeded, codes.Aborted:
        return true
    }
    return false
}

err := client.ControllerExpandVolume(ctx, req)
if err != nil {
    if isRetryableGRPCError(err) {
        // retry with backoff
    } else if strings.Contains(err.Error(), "OutOfRange") {
        // capacity range unsupported: adjust size
    }
    return err
}

When it happens

Trigger: Calling ControllerExpandVolume with a volume ID and capacity range when the plugin responds with a gRPC status other than OutOfRange or Internal: e.g. the plugin is temporarily unavailable (codes.Unavailable), the ExternalVolumeID does not exist in the plugin (codes.NotFound), the plugin does not implement ControllerExpandVolume (codes.Unimplemented), or the RPC times out (codes.DeadlineExceeded).

Common situations: Expanding a volume whose plugin crashed or restarted mid-request; typos or stale volume IDs after re-provisioning storage out-of-band; running a plugin image that lacks ControllerExpandVolume support (older CSI spec version); network partition between Nomad client and the plugin unix socket/TCP endpoint.

Related errors

• volume %q cannot be expanded while in use: %v
• nil response from plugin.NodeExpandVolume
• expand is not implemented by this controller plugin
• one of LimitBytes or RequiredBytes must be set
• CSI.ControllerExpandVolume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0ebc38d38bf99707. Report an issue: GitHub.