{"record":{"id":"31b4d17a13b09569","repo":"hashicorp/nomad","slug":"error-getting-plugin-s-v","errorCode":null,"errorMessage":"error getting plugin: %s, %v","messagePattern":"error getting plugin: (.+?), (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_csi_endpoint.go","lineNumber":303,"sourceCode":"func (a *ClientCSI) clientIDsForController(pluginID string) ([]string, error) {\n\n\tsnap, err := a.srv.State().Snapshot()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif pluginID == \"\" {\n\t\treturn nil, fmt.Errorf(\"missing plugin ID\")\n\t}\n\n\tws := memdb.NewWatchSet()\n\n\t// note: plugin IDs are not scoped to region but volumes are. so any Nomad\n\t// client we get for a controller is already in the same region for the\n\t// volume.\n\tplugin, err := snap.CSIPluginByID(ws, pluginID)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error getting plugin: %s, %v\", pluginID, err)\n\t}\n\tif plugin == nil {\n\t\treturn nil, fmt.Errorf(\"plugin missing: %s\", pluginID)\n\t}\n\n\tclientIDs := []string{}\n\n\tif len(plugin.Controllers) == 0 {\n\t\treturn nil, fmt.Errorf(\"failed to find instances of controller plugin %q\", pluginID)\n\t}\n\n\tvar merr error\n\tfor clientID, controller := range plugin.Controllers {\n\t\tif !controller.IsController() {\n\t\t\t// we don't have separate types for CSIInfo depending on whether\n\t\t\t// it's a controller or node. this error should never make it to\n\t\t\t// production\n\t\t\tmerr = errors.Join(merr, fmt.Errorf(","sourceCodeStart":285,"sourceCodeEnd":321,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_csi_endpoint.go#L285-L321","documentation":"clientIDsForController calls snap.CSIPluginByID(ws, pluginID); if the state store lookup returns a non-nil error it is wrapped as 'error getting plugin: <id>, <v>'. This indicates a state-store failure while reading the CSI plugin registry, not merely an absent plugin (that yields 'plugin missing').","triggerScenarios":"sendCSIControllerRPC → clientIDsForController when the state snapshot's CSIPluginByID query errors — e.g. state store backend failure, snapshot inconsistency, or memdb error during server startup/shutdown.","commonSituations":"Nomad server state store degraded (raft issues, shutdown in progress); corrupted or migrating state store; transient errors during leader failover.","solutions":["Check Nomad server logs and raft/leader health (`nomad server members`, leader election state).","Retry the operation after the cluster stabilizes.","If errors persist, investigate state store integrity (server data dir, restore from backup per HashiCorp guidance)."],"exampleFix":"// operator check before retrying\nnomad server members   # ensure a stable leader\n# after leader stabilizes\nnomad volume snapshots -plugin aws-ebs-controller","handlingStrategy":"retry","validationCode":"// preflight: check server reachability/leader before state-store reads\nleader, _, err := client.Status().Leader()\nif err != nil || leader == \"\" { return fmt.Errorf(\"no stable leader; defer RPCs\") }","typeGuard":"func clusterReady(leader string) bool { return leader != \"\" }","tryCatchPattern":"err := csi.ControllerListVolumes(args, reply)\nif err != nil && strings.Contains(err.Error(), \"error getting plugin:\") {\n    // transient state-store issue: backoff and retry\n    time.Sleep(backoff)\n}","preventionTips":["Retry with backoff around leader elections/failovers.","Monitor Nomad server raft health.","Avoid CSI RPCs during server maintenance windows.","Escalate to state-store integrity checks if persistent."],"tags":["csi","nomad","state-store","internal"],"backgroundTag":"state-store-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}