{"record":{"id":"b96d97bdd302a1ae","repo":"hashicorp/nomad","slug":"csi-controllerexpandvolume-could-not-find-plugin","errorCode":null,"errorMessage":"CSI.ControllerExpandVolume could not find plugin: %w: %v (wraps ErrCSIClientRPCRetryable)","messagePattern":"CSI\\.ControllerExpandVolume could not find plugin: %w: (.+?) \\(wraps ErrCSIClientRPCRetryable\\)","errorType":"exception","errorClass":"ErrCSIClientRPCRetryable","httpStatus":null,"severity":"warning","filePath":"client/csi_endpoint.go","lineNumber":243,"sourceCode":"\t// the server RPC call\n\n\tresp.Topologies = make([]*nstructs.CSITopology, len(cresp.Volume.AccessibleTopology))\n\tfor _, topo := range cresp.Volume.AccessibleTopology {\n\t\tresp.Topologies = append(resp.Topologies,\n\t\t\t&nstructs.CSITopology{Segments: topo.Segments})\n\t}\n\n\treturn nil\n}\n\nfunc (c *CSI) ControllerExpandVolume(req *structs.ClientCSIControllerExpandVolumeRequest, resp *structs.ClientCSIControllerExpandVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"expand_volume\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerExpandVolume could not find plugin: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq := req.ToCSIRequest()\n\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\n\t// CSI ControllerExpandVolume errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\tcresp, err := plugin.ControllerExpandVolume(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {\n\t\t// if the volume was deleted out-of-band, we'll get an error from\n\t\t// the plugin but can safely ignore it","sourceCodeStart":225,"sourceCodeEnd":261,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L225-L261","documentation":"Returned by the CSI ControllerExpandVolume handler when c.findControllerPlugin(req.PluginID) cannot locate a running controller plugin instance on this client. The error wraps nstructs.ErrCSIClientRPCRetryable, signalling the Nomad server that its view of plugin placement/health is stale and the RPC should be retried against another client (or after the plugin registers).","triggerScenarios":"findControllerPlugin fails because the plugin ID in the volume's request is not registered on this node, the controller plugin task has crashed/stopped, or the plugin has not yet finished fingerprinting/registration after client or node startup.","commonSituations":"CSI controller plugin job stopped or OOM-killed; volume registered with a pluginID typo; node running only the node-plugin (no controller); transient state during cluster startup or plugin upgrade; server routing the expand request to a client that no longer hosts the controller.","solutions":["Retry the volume expansion — the error is explicitly marked retryable and the server should re-route to a healthy controller instance","Check the controller plugin job is running on some node (nomad job status <csi-plugin-job>) and restart it if it crashed","Verify the pluginID on the volume registration matches the registered CSI plugin (nomad plugin status)","Wait for plugin fingerprinting to complete after a node restart before issuing expansions","Confirm the controller plugin (not just node plugin) is deployed for the storage backend"],"exampleFix":"// before: expanding immediately after plugin upgrade\nnomad volume detach/expand ...\n// after: verify controller health first, then retry on this error\nnomad plugin status <plugin-id>\n# if no controllers healthy: nomad job start <csi-controller-job>, then retry expand","handlingStrategy":"retry","validationCode":"// Before expanding, confirm the controller plugin is registered and healthy\n// API: GET /v1/volumes/csi/<id> -> check ControllersHealthy > 0\n// shell: nomad plugin status <plugin-id>","typeGuard":null,"tryCatchPattern":"// The error wraps ErrCSIClientRPCRetryable; detect and retry with backoff\nerr := client.ExpandVolume(req)\nif err != nil && strings.Contains(err.Error(), \"could not find plugin\") {\n    time.Sleep(backoff) // server will re-route to another client instance\n    err = client.ExpandVolume(req)\n}","preventionTips":["Run the CSI controller plugin as a Nomad job with restart/reschedule stanzas so it self-heals","Ensure the controller plugin is deployed cluster-wide or with affinity to nodes needing it","Double-check pluginID spelling on volume registrations","After node/plugin upgrades, wait for fingerprinting before issuing controller RPCs"],"tags":["csi","plugin-not-found","retryable","volume-expansion"],"backgroundTag":"csi-plugin-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}