{"record":{"id":"0ebc38d38bf99707","repo":"hashicorp/nomad","slug":"controller-plugin-returned-an-error-v","errorCode":null,"errorMessage":"controller plugin returned an error: %v","messagePattern":"controller plugin returned an error: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"plugins/csi/client.go","lineNumber":544,"sourceCode":"\tresp, err := c.controllerClient.ControllerExpandVolume(ctx, exReq, opts...)\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.InvalidArgument:\n\t\t\treturn nil, fmt.Errorf(\n\t\t\t\t\"requested capabilities not compatible with volume %q: %v\",\n\t\t\t\treq.ExternalVolumeID, err)\n\t\tcase codes.NotFound:\n\t\t\terr = fmt.Errorf(\"volume %q could not be found: %v\", req.ExternalVolumeID, err)\n\t\tcase codes.FailedPrecondition:\n\t\t\terr = fmt.Errorf(\"volume %q cannot be expanded online: %v\", req.ExternalVolumeID, err)\n\t\tcase codes.OutOfRange:\n\t\t\treturn nil, fmt.Errorf(\n\t\t\t\t\"unsupported capacity_range for volume %q: %v\", req.ExternalVolumeID, err)\n\t\tcase codes.Internal:\n\t\t\terr = fmt.Errorf(\"controller plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\tdefault:\n\t\t\terr = fmt.Errorf(\"controller plugin returned an error: %v\", err)\n\t\t}\n\t\treturn nil, err\n\t}\n\n\treturn &ControllerExpandVolumeResponse{\n\t\tCapacityBytes:         resp.GetCapacityBytes(),\n\t\tNodeExpansionRequired: resp.GetNodeExpansionRequired(),\n\t}, nil\n}\n\n// compareCapabilities returns an error if the 'got' capabilities aren't found\n// within the 'expected' capability.\n//\n// Note that plugins in the wild are known to return incomplete\n// VolumeCapability responses, so we can't require that all capabilities we\n// expect have been validated, only that the ones that have been validated\n// match. This appears to violate the CSI specification but until that's been\n// resolved in upstream we have to loosen our validation requirements. The","sourceCodeStart":526,"sourceCodeEnd":562,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L526-L562","documentation":"This error wraps any gRPC status error returned by the CSI controller plugin during ControllerExpandVolume that is not OutOfRange or Internal (e.g. Unavailable, DeadlineExceeded, NotFound, Unimplemented). Nomad converts the plugin's gRPC error into a generic message prefixed with 'controller plugin returned an error' so the caller knows the failure originated in the external storage plugin, not Nomad itself.","triggerScenarios":"Calling ControllerExpandVolume with a volume ID and capacity range when the plugin responds with a gRPC status other than OutOfRange or Internal: e.g. the plugin is temporarily unavailable (codes.Unavailable), the ExternalVolumeID does not exist in the plugin (codes.NotFound), the plugin does not implement ControllerExpandVolume (codes.Unimplemented), or the RPC times out (codes.DeadlineExceeded).","commonSituations":"Expanding a volume whose plugin crashed or restarted mid-request; typos or stale volume IDs after re-provisioning storage out-of-band; running a plugin image that lacks ControllerExpandVolume support (older CSI spec version); network partition between Nomad client and the plugin unix socket/TCP endpoint.","solutions":["Read the %v suffix of the error for the underlying gRPC status and check the plugin's own logs for the root cause","Verify the ExternalVolumeID exists in the storage backend and matches what the plugin knows","Check plugin container health/restart it if the status is Unavailable or DeadlineExceeded","Confirm the plugin implements the CSI 1.x ControllerExpandVolume RPC (not a 'controller capabilities' limited plugin)","Retry the expansion if the status is transient (Unavailable/DeadlineExceeded)"],"exampleFix":"// before: expanding a volume whose ID no longer exists in the backend\nclient.ControllerExpandVolume(ctx, &csi.ControllerExpandVolumeRequest{ExternalVolumeID: \"vol-stale-id\", CapacityRange: ...})\n// error: controller plugin returned an error: rpc error: code = NotFound ...\n\n// after: verify the volume ID against the backend first, or list volumes via ControllerGetCapabilities/ListVolumes before expanding","handlingStrategy":"retry","validationCode":"// before expanding, confirm plugin is healthy and volume exists\nresp, err := plugin.ControllerGetCapabilities(ctx, &csi.ControllerGetCapabilitiesRequest{})\nif err != nil { return fmt.Errorf(\"plugin controller unavailable: %w\", err) }\nvar hasExpand bool\nfor _, c := range resp.GetCapabilities() {\n    if c.GetRpc().GetType() == csipbv1.ControllerServiceCapability_RPC_EXPAND_VOLUME { hasExpand = true }\n}\nif !hasExpand { return errors.New(\"plugin does not support ControllerExpandVolume\") }","typeGuard":"func isRetryableGRPCError(err error) bool {\n    st, ok := status.FromError(err)\n    if !ok { return false }\n    switch st.Code() {\n    case codes.Unavailable, codes.DeadlineExceeded, codes.Aborted:\n        return true\n    }\n    return false\n}","tryCatchPattern":"err := client.ControllerExpandVolume(ctx, req)\nif err != nil {\n    if isRetryableGRPCError(err) {\n        // retry with backoff\n    } else if strings.Contains(err.Error(), \"OutOfRange\") {\n        // capacity range unsupported: adjust size\n    }\n    return err\n}","preventionTips":["Check plugin controller capabilities (EXPAND_VOLUME) before attempting expansion","Verify the ExternalVolumeID exists in the storage backend before expanding","Monitor plugin container health and socket connectivity from the Nomad client","Wrap expansion calls in bounded retry with backoff for transient gRPC codes","Keep the CSI plugin version aligned with the CSI spec features you use"],"tags":["csi","grpc","storage","volume-expansion"],"backgroundTag":"csi-controller-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}