{"record":{"id":"19b0e94cb7734906","repo":"hashicorp/nomad","slug":"plugin-lookup-error-s-v","errorCode":null,"errorMessage":"plugin lookup error: %s %v","messagePattern":"plugin lookup error: (.+?) (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3030,"sourceCode":"// allocations.\nfunc (s *StateStore) CSIVolumeDenormalizePlugins(ws memdb.WatchSet, vol *structs.CSIVolume) (*structs.CSIVolume, error) {\n\tif vol == nil {\n\t\treturn nil, nil\n\t}\n\ttxn := s.db.ReadTxn()\n\tdefer txn.Abort()\n\treturn s.csiVolumeDenormalizePluginsTxn(txn, vol)\n}\n\n// csiVolumeDenormalizePluginsTxn implements\n// CSIVolumeDenormalizePlugins, inside a transaction.\nfunc (s *StateStore) csiVolumeDenormalizePluginsTxn(txn Txn, vol *structs.CSIVolume) (*structs.CSIVolume, error) {\n\tif vol == nil {\n\t\treturn nil, nil\n\t}\n\tplug, err := s.CSIPluginByIDTxn(txn, nil, vol.PluginID)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"plugin lookup error: %s %v\", vol.PluginID, err)\n\t}\n\tif plug == nil {\n\t\tvol.ControllersHealthy = 0\n\t\tvol.NodesHealthy = 0\n\t\tvol.Schedulable = false\n\t\treturn vol, nil\n\t}\n\n\tvol.Provider = plug.Provider\n\tvol.ProviderVersion = plug.Version\n\tvol.ControllerRequired = plug.ControllerRequired\n\tvol.ControllersHealthy = plug.ControllersHealthy\n\tvol.NodesHealthy = plug.NodesHealthy\n\n\t// This value may be stale, but stale is ok\n\tvol.ControllersExpected = plug.ControllersExpected\n\tvol.NodesExpected = plug.NodesExpected\n","sourceCodeStart":3012,"sourceCodeEnd":3048,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3012-L3048","documentation":"csiVolumeDenormalizePluginsTxn enriches a CSIVolume with health/schedulability data from its plugin. When CSIPluginByIDTxn returns a non-nil error (a memdb lookup failure on the csi_plugins table, not merely a missing plugin), this wrapper propagates it prefixed with the volume's PluginID. Note a nil plugin with nil error is handled gracefully (health zeroed), so this error only fires on genuine lookup failures.","triggerScenarios":"Reading a CSIVolume (CSIVolumeByID / volume list denormalization) whose PluginID causes CSIPluginByIDTxn's txn.FirstWatch on csi_plugins to return an error — memdb index failure, not a missing row.","commonSituations":"State store index corruption, version-skewed state database, or bugs in custom forked state store code with altered csi_plugins indexes.","solutions":["Restart the server agent to rebuild memdb state from Raft.","Check the wrapped inner error (%v) for the actual memdb failure and address it.","Ensure the plugin row exists via nomad plugin status / CSIPluginByID; re-register the plugin if needed.","Align Nomad versions across the cluster."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// before denormalizing, confirm plugin exists:\nplug, err := store.CSIPluginByID(ws, ns, vol.PluginID)\nif err != nil { return nil, err }\nif plug == nil { vol.Schedulable = false } // missing plugin is not an error","typeGuard":"func pluginUsable(p *structs.CSIPlugin) bool {\n    return p != nil && p.ControllersHealthy > 0\n}","tryCatchPattern":"vol, err := store.CSIVolumeByID(ws, ns, id)\nif err != nil && strings.Contains(err.Error(), \"plugin lookup error\") {\n    log.Warn(\"plugin denormalization failed; serving un-enriched volume\", \"err\", err)\n    return vol, nil // degrade gracefully\n}","preventionTips":["Handle nil-plugin results separately from errors — absence is normal","Watch CSI plugin deregistration events (nomad plugin status) to anticipate denormalize failures","Restart state store on any repeated lookup errors","Pin cluster to a single Nomad version"],"tags":["nomad","state-store","csi","plugin-lookup"],"backgroundTag":"csi-plugin-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}