hashicorp/nomad · error

unable to expand volume: %w

Error message

unable to expand volume: %w

What it means

Wrap-around error from expandVolume: Nomad forwarded the ControllerExpandVolume RPC to the CSI controller plugin and the RPC failed. The underlying cause (plugin error, cloud API failure, timeout, permission denial) is wrapped with %w so errors.Is/As work; this error means the expansion was not confirmed and vol.Capacity was not updated.

Source

Thrown at nomad/csi_endpoint.go:1376

	method := "ClientCSI.ControllerExpandVolume"
	cReq := &cstructs.ClientCSIControllerExpandVolumeRequest{
		ExternalVolumeID: vol.ExternalID,
		Secrets:          vol.Secrets,
		CapacityRange:    capacity,
		VolumeCapability: capability,
	}
	cReq.PluginID = plugin.ID
	cResp := &cstructs.ClientCSIControllerExpandVolumeResponse{}

	logger.Info("starting volume expansion")
	// This is the real work. The client RPC sends a gRPC to the controller plugin,
	// then that controller may reach out to cloud APIs, etc.
	err = v.serializedControllerRPC(plugin.ID, func() error {
		return v.srv.RPC(method, cReq, cResp)
	})
	if err != nil {
		return fmt.Errorf("unable to expand volume: %w", err)
	}
	vol.Capacity = cResp.CapacityBytes
	logger.Info("controller done expanding volume")

	if cResp.NodeExpansionRequired {
		return v.nodeExpandVolume(vol, plugin, capacity)
	}

	return nil
}

// nodeExpandVolume sends NodeExpandVolume requests to the appropriate client
// for each allocation that has a claim on the volume. The client will then
// send a gRPC call to the CSI node plugin colocated with the allocation.
func (v *CSIVolume) nodeExpandVolume(vol *structs.CSIVolume, plugin *structs.CSIPlugin, capacity *csi.CapacityRange) error {
	var mErr multierror.Error
	logger := v.logger.Named("nodeExpandVolume").
		With("volume", vol.ID, "plugin", plugin.ID)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause and plugin logs (`nomad volume status`, `nomad monitor-log`, plugin stderr) to find the real failure from the controller plugin.
2. Ensure the CSI controller plugin is running and healthy (`nomad plugin status <plugin-id>`) and retry the expand once healthy.
3. Check the storage backend: free quota/capacity limits, volume state (in-use vs available), and whether online expansion is supported.
4. Retry after resolving transient backend errors; if a node expansion was required and controller succeeded, verify nodes picked up the resize.

p, err := client.Plugins().Get(pluginID)
if err != nil || p.ControllersHealthy == 0 {
    return fmt.Errorf("CSI controller for %s not healthy; fix before expanding", pluginID)
}

func isExpandRPCError(err error) bool {
    return err != nil && strings.Contains(err.Error(), "unable to expand volume")
}

err := client.Volumes().Expand(id, min, max)
if err != nil {
    var rpcErr *timeouts.Error
    if errors.As(errors.UnwrapAll(err), &rpcErr) || isTransient(err) {
        // backoff and retry once controller is healthy
    }
    log.Fatalf("expand failed: %v", err) // wrapped cause printed via %v/%+v
}

When it happens

Trigger: Running `nomad volume expand` when the CSI controller plugin is not running/healthy, the plugin returns a gRPC error, the storage backend rejects the expansion (quota, capacity exhausted, volume in use), or the serialized controller RPC times out.

Common situations: Controller plugin task crashed or node running it is down; storage provider quota/limit hit; volume attached to a node that does not support online expansion; ACL/plugin misconfiguration; transient cloud API outage during controller expansion.

Related errors

• CSI Plugin loaded incorrectly
• CSI.ControllerListVolumes: plugin returned an invalid entry
• CSI.NodeDetachVolume: %v
• controller attach volume: %v
• controller validate volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e42ce4764f49ce52. Report an issue: GitHub.