{"record":{"id":"e42ce4764f49ce52","repo":"hashicorp/nomad","slug":"unable-to-expand-volume-w","errorCode":null,"errorMessage":"unable to expand volume: %w","messagePattern":"unable to expand volume: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/csi_endpoint.go","lineNumber":1376,"sourceCode":"\n\tmethod := \"ClientCSI.ControllerExpandVolume\"\n\tcReq := &cstructs.ClientCSIControllerExpandVolumeRequest{\n\t\tExternalVolumeID: vol.ExternalID,\n\t\tSecrets:          vol.Secrets,\n\t\tCapacityRange:    capacity,\n\t\tVolumeCapability: capability,\n\t}\n\tcReq.PluginID = plugin.ID\n\tcResp := &cstructs.ClientCSIControllerExpandVolumeResponse{}\n\n\tlogger.Info(\"starting volume expansion\")\n\t// This is the real work. The client RPC sends a gRPC to the controller plugin,\n\t// then that controller may reach out to cloud APIs, etc.\n\terr = v.serializedControllerRPC(plugin.ID, func() error {\n\t\treturn v.srv.RPC(method, cReq, cResp)\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"unable to expand volume: %w\", err)\n\t}\n\tvol.Capacity = cResp.CapacityBytes\n\tlogger.Info(\"controller done expanding volume\")\n\n\tif cResp.NodeExpansionRequired {\n\t\treturn v.nodeExpandVolume(vol, plugin, capacity)\n\t}\n\n\treturn nil\n}\n\n// nodeExpandVolume sends NodeExpandVolume requests to the appropriate client\n// for each allocation that has a claim on the volume. The client will then\n// send a gRPC call to the CSI node plugin colocated with the allocation.\nfunc (v *CSIVolume) nodeExpandVolume(vol *structs.CSIVolume, plugin *structs.CSIPlugin, capacity *csi.CapacityRange) error {\n\tvar mErr multierror.Error\n\tlogger := v.logger.Named(\"nodeExpandVolume\").\n\t\tWith(\"volume\", vol.ID, \"plugin\", plugin.ID)","sourceCodeStart":1358,"sourceCodeEnd":1394,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L1358-L1394","documentation":"Wrap-around error from expandVolume: Nomad forwarded the ControllerExpandVolume RPC to the CSI controller plugin and the RPC failed. The underlying cause (plugin error, cloud API failure, timeout, permission denial) is wrapped with %w so errors.Is/As work; this error means the expansion was not confirmed and vol.Capacity was not updated.","triggerScenarios":"Running `nomad volume expand` when the CSI controller plugin is not running/healthy, the plugin returns a gRPC error, the storage backend rejects the expansion (quota, capacity exhausted, volume in use), or the serialized controller RPC times out.","commonSituations":"Controller plugin task crashed or node running it is down; storage provider quota/limit hit; volume attached to a node that does not support online expansion; ACL/plugin misconfiguration; transient cloud API outage during controller expansion.","solutions":["Inspect the wrapped cause and plugin logs (`nomad volume status`, `nomad monitor-log`, plugin stderr) to find the real failure from the controller plugin.","Ensure the CSI controller plugin is running and healthy (`nomad plugin status <plugin-id>`) and retry the expand once healthy.","Check the storage backend: free quota/capacity limits, volume state (in-use vs available), and whether online expansion is supported.","Retry after resolving transient backend errors; if a node expansion was required and controller succeeded, verify nodes picked up the resize."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"p, err := client.Plugins().Get(pluginID)\nif err != nil || p.ControllersHealthy == 0 {\n    return fmt.Errorf(\"CSI controller for %s not healthy; fix before expanding\", pluginID)\n}","typeGuard":"func isExpandRPCError(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"unable to expand volume\")\n}","tryCatchPattern":"err := client.Volumes().Expand(id, min, max)\nif err != nil {\n    var rpcErr *timeouts.Error\n    if errors.As(errors.UnwrapAll(err), &rpcErr) || isTransient(err) {\n        // backoff and retry once controller is healthy\n    }\n    log.Fatalf(\"expand failed: %v\", err) // wrapped cause printed via %v/%+v\n}","preventionTips":["Check `nomad plugin status <plugin-id>` controller health before expanding.","Monitor storage backend quotas and provider status during expansion.","Keep plugin job replicas healthy; avoid expanding during node drains.","Use errors.Is/As on the wrapped cause to distinguish retryable vs fatal failures."],"tags":["csi","rpc","plugin","storage"],"backgroundTag":"csi-controller-rpc-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}