{"record":{"id":"47a5328e0b8e673e","repo":"hashicorp/nomad","slug":"failed-to-find-instances-of-controller-plugin-q","errorCode":null,"errorMessage":"failed to find instances of controller plugin %q","messagePattern":"failed to find instances of controller plugin %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":312,"sourceCode":"\t}\n\n\tws := memdb.NewWatchSet()\n\n\t// note: plugin IDs are not scoped to region but volumes are. so any Nomad\n\t// client we get for a controller is already in the same region for the\n\t// volume.\n\tplugin, err := snap.CSIPluginByID(ws, pluginID)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error getting plugin: %s, %v\", pluginID, err)\n\t}\n\tif plugin == nil {\n\t\treturn nil, fmt.Errorf(\"plugin missing: %s\", pluginID)\n\t}\n\n\tclientIDs := []string{}\n\n\tif len(plugin.Controllers) == 0 {\n\t\treturn nil, fmt.Errorf(\"failed to find instances of controller plugin %q\", pluginID)\n\t}\n\n\tvar merr error\n\tfor clientID, controller := range plugin.Controllers {\n\t\tif !controller.IsController() {\n\t\t\t// we don't have separate types for CSIInfo depending on whether\n\t\t\t// it's a controller or node. this error should never make it to\n\t\t\t// production\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"plugin instance %q is not a controller but was registered as one - this is always a bug\", controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif !controller.Healthy {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"plugin instance %q is not healthy\", controller.AllocID))\n\t\t\tcontinue\n\t\t}","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L294-L330","documentation":"If the CSI plugin record exists but its Controllers map is empty, clientIDsForController returns 'failed to find instances of controller plugin %q'. The plugin is registered but no controller-capable instance has ever reported to it, so there is nowhere to forward the controller RPC.","triggerScenarios":"Controller RPC sent when the controller plugin job has zero running allocations (controller-only job dead, or only node plugins running for a plugin that also needs a controller).","commonSituations":"Controller plugin job stopped/scaled to zero; plugin job failing to schedule (unsatisfiable constraints, no eligible clients); a node-only CSI plugin mistakenly used as a controller target.","solutions":["Check the controller plugin job: `nomad job status <csi-plugin-job>` and fix scheduling failures or restart it.","Verify node constraints/CSIPluginID placement so the controller can be scheduled.","Confirm the driver actually ships a controller component (some drivers are node-only).","Once `nomad plugin status <id>` shows controller instances healthy, retry."],"exampleFix":"# before: controller count 0\nnomad job stop aws-ebs-csi-controller\n# after\nnomad job run aws-ebs-csi-controller.nomad.hcl\nnomad plugin status aws-ebs-controller  # Controllers: 1/1 healthy","handlingStrategy":"validation","validationCode":"p, _, _ := client.Plugins().Info(ctx, pluginID, nil)\nif p != nil && p.ControllersExpected > 0 && p.ControllersHealthy == 0 {\n    return fmt.Errorf(\"controller plugin %q has no running instances\", pluginID)\n}","typeGuard":"func hasControllerInstances(p *api.CSIPlugin) bool {\n    return p != nil && p.ControllersHealthy > 0\n}","tryCatchPattern":"err := csi.ControllerListVolumes(args, reply)\nif err != nil && strings.Contains(err.Error(), \"failed to find instances of controller plugin\") {\n    return fmt.Errorf(\"start the controller plugin job: %w\", err)\n}","preventionTips":["Run the controller job before issuing controller RPCs.","Add restart stanza + health checks to plugin jobs.","Verify driver provides a controller component, not node-only.","Alert when ControllersHealthy drops to zero."],"tags":["csi","nomad","plugin-registration","controller"],"backgroundTag":"csi-plugin-not-registered","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}