{"record":{"id":"da5e996c2d96aa9b","repo":"hashicorp/nomad","slug":"plugin-instance-q-is-not-healthy","errorCode":null,"errorMessage":"plugin instance %q is not healthy","messagePattern":"plugin instance %q is not healthy","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":327,"sourceCode":"\tclientIDs := []string{}\n\n\tif len(plugin.Controllers) == 0 {\n\t\treturn nil, fmt.Errorf(\"failed to find instances of controller plugin %q\", pluginID)\n\t}\n\n\tvar merr error\n\tfor clientID, controller := range plugin.Controllers {\n\t\tif !controller.IsController() {\n\t\t\t// we don't have separate types for CSIInfo depending on whether\n\t\t\t// it's a controller or node. this error should never make it to\n\t\t\t// production\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"plugin instance %q is not a controller but was registered as one - this is always a bug\", controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif !controller.Healthy {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"plugin instance %q is not healthy\", controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tnode, err := getNodeForRpc(snap, clientID)\n\t\tif err != nil || node == nil {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"cannot find node %q for plugin instance %q\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tif node.Status != structs.NodeStatusReady {\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(\n\t\t\t\t\"node %q for plugin instance %q is not ready\", clientID, controller.AllocID))\n\t\t\tcontinue\n\t\t}\n\n\t\tclientIDs = append(clientIDs, clientID)","sourceCodeStart":309,"sourceCodeEnd":345,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L309-L345","documentation":"clientIDsForController skips controller instances whose CSIInfo.Healthy is false, joining 'plugin instance %q is not healthy' into a combined error. If all instances are skipped, the caller has no healthy node to forward the controller RPC to and the joined error surfaces.","triggerScenarios":"All controller plugin instances on the plugin record have Healthy=false — plugin allocations crashed, failed health checks, or clients stopped heartbeating plugin info after a node loss.","commonSituations":"Controller alloc OOMing or crashing-looping; nomad client agent down so plugin heartbeats lapse; plugin failed CSI probe after backend credential changes; node drained/terminated while plugin record persists.","solutions":["Inspect `nomad plugin status <pluginID>` for instance health and the alloc logs of the controller job (`nomad alloc logs <alloc>`).","Fix the root cause (OOM, bad credentials, driver probe failure) and redeploy the plugin job.","Check `nomad node status` for lost/down clients and restore client agent connectivity so heartbeats resume.","Retry once at least one controller instance reports healthy."],"exampleFix":"# before: controller alloc crash-looping on bad secret\n# aws_secret_access_key = \"stale\"\n# after: update secrets and reschedule\nnomad job run -vault-namespace=csi aws-ebs-csi-controller.nomad.hcl\nnomad plugin status aws-ebs-controller  # instances healthy","handlingStrategy":"retry","validationCode":"p, _, _ := client.Plugins().Info(ctx, pluginID, nil)\nif p.ControllersHealthy == 0 {\n    return fmt.Errorf(\"wait: all %d controller instances unhealthy\", p.ControllersExpected)\n}","typeGuard":"func anyHealthyController(p *api.CSIPlugin) bool { return p.ControllersHealthy > 0 }","tryCatchPattern":"err := csi.ControllerListVolumes(args, reply)\nif err != nil && strings.Contains(err.Error(), \"is not healthy\") {\n    // joined error may list several allocs; inspect plugin status then retry\n    return retryAfterPluginRecovers(err)\n}","preventionTips":["Monitor CSI plugin instance health and alloc crash loops.","Fix credential/probe failures that flip instances unhealthy.","Ensure client agents stay connected so plugin heartbeats don't lapse.","Retry after `nomad plugin status` reports instances healthy again."],"tags":["csi","nomad","plugin-health","heartbeat"],"backgroundTag":"csi-plugin-unhealthy","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}