hashicorp/nomad · error

CSI.ControllerExpandVolume: plugin did not return error or r

Error message

CSI.ControllerExpandVolume: plugin did not return error or response

What it means

In Nomad's CSI client endpoint, after calling the CSI plugin's ControllerExpandVolume RPC, the plugin returned neither an error nor a response object (cresp == nil). The gRPC/Go CSI plugin contract requires one of the two, so this indicates a malformed plugin implementation. Nomad explicitly logs it as a bug in the plugin that should be reported to the plugin author.

Source

Thrown at client/csi_endpoint.go:270

	// CSI ControllerExpandVolume errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	cresp, err := plugin.ControllerExpandVolume(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the volume was deleted out-of-band, we'll get an error from
		// the plugin but can safely ignore it
		c.c.logger.Debug("could not expand volume", "error", err)
		return nil
	}
	if err != nil {
		return fmt.Errorf("CSI.ControllerExpandVolume: %v", err)
	}
	if cresp == nil {
		c.c.logger.Warn("plugin did not return error or response; this is a bug in the plugin and should be reported to the plugin author")
		return fmt.Errorf("CSI.ControllerExpandVolume: plugin did not return error or response")
	}
	resp.CapacityBytes = cresp.CapacityBytes
	resp.NodeExpansionRequired = cresp.NodeExpansionRequired
	return nil
}

func (c *CSI) ControllerDeleteVolume(req *structs.ClientCSIControllerDeleteVolumeRequest, resp *structs.ClientCSIControllerDeleteVolumeResponse) error {
	defer metrics.MeasureSince([]string{"client", "csi_controller", "delete_volume"}, time.Now())

	plugin, err := c.findControllerPlugin(req.PluginID)
	if err != nil {
		// the server's view of the plugin health is stale, so let it know it
		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerDeleteVolume: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Report the bug to the CSI plugin vendor/author; the plugin must return either a non-nil ControllerExpandVolumeResponse or a non-nil error
2. Upgrade the CSI plugin to the latest version where the nil-response bug may be fixed
3. Check Nomad and plugin version compatibility and restart the plugin task; re-run the volume expansion

Example fix

// before (buggy plugin code)
func (p *Plugin) ControllerExpandVolume(ctx context.Context, req *csi.ControllerExpandVolumeRequest) (*csi.ControllerExpandVolumeResponse, error) {
    return nil, nil
}
// after
func (p *Plugin) ControllerExpandVolume(ctx context.Context, req *csi.ControllerExpandVolumeRequest) (*csi.ControllerExpandVolumeResponse, error) {
    resp, err := p.backend.Expand(req.VolumeId, req.CapacityRange.RequiredBytes)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "expand failed: %v", err)
    }
    return &csi.ControllerExpandVolumeResponse{CapacityBytes: resp.Capacity, NodeExpansionRequired: true}, nil
}

// before relying on expand, verify plugin health and implementation
plug, err := apiClient.Plugins().Get("aws-ebs-controller")
if err != nil || plug.ControllersHealthy < 1 {
    log.Fatal("controller plugin not healthy; expand would fail")
}
if plug.ControllerInfo.SupportsCondition(structs.CSIControllerSupportsExpandVolume) == false {
    log.Fatal("plugin does not advertise expand support")
}

func isNilPluginResponseError(err error) bool {
    return err != nil && strings.Contains(err.Error(),
        "plugin did not return error or response")
}

err := client.CSI().ControllerExpandVolume(req)
if err != nil {
    if isNilPluginResponseError(err) {
        // not a transient storage failure: report the plugin bug, don't retry
        reportPluginBug(req.PluginID, err)
        return err
    }
    return err
}

When it happens

Trigger: A CSI controller plugin (or its client wrapper) returns (nil, nil) from ControllerExpandVolume — e.g. a buggy custom/plugin-in-development implementation, a plugin that silently swallows errors and skips building a response, or a hand-rolled CSI mock plugin.

Common situations: Developers testing a custom CSI plugin against Nomad, running an outdated/buggy plugin version that mishandles the expand RPC, or using a stub plugin that returns empty results on ControllerExpandVolume.

Related errors

• CSI plugin failed to register: %w
• failed to probe plugin: %w
• CSI.ControllerDeleteVolume: %v
• CSI.ControllerListVolumes: %v
• %v: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ed6d0e8c6ebd2760. Report an issue: GitHub.