{"record":{"id":"46641f4243bc3ec8","repo":"hashicorp/nomad","slug":"csi-plugin-lookup-error-s-v","errorCode":null,"errorMessage":"csi_plugin lookup error: %s %v","messagePattern":"csi_plugin lookup error: (.+?) (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1415,"sourceCode":"\t\te.CreateIndex = index\n\t\tnode.Events = append(node.Events, e)\n\t}\n\n\t// Keep node events pruned to not exceed the max allowed\n\tif l := len(node.Events); l > structs.MaxRetainedNodeEvents {\n\t\tdelta := l - structs.MaxRetainedNodeEvents\n\t\tnode.Events = node.Events[delta:]\n\t}\n}\n\n// upsertCSIPluginsForNode indexes csi plugins for volume retrieval, with health. It's called\n// on upsertNodeEvents, so that event driven health changes are updated\nfunc upsertCSIPluginsForNode(txn *txn, node *structs.Node, index uint64) error {\n\n\tupsertFn := func(info *structs.CSIInfo) error {\n\t\traw, err := txn.First(TableCSIPlugins, \"id\", info.PluginID)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"csi_plugin lookup error: %s %v\", info.PluginID, err)\n\t\t}\n\n\t\tvar plug *structs.CSIPlugin\n\t\tif raw != nil {\n\t\t\tplug = raw.(*structs.CSIPlugin).Copy()\n\t\t} else {\n\t\t\tif !info.Healthy {\n\t\t\t\t// we don't want to create new plugins for unhealthy\n\t\t\t\t// allocs, otherwise we'd recreate the plugin when we\n\t\t\t\t// get the update for the alloc becoming terminal\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\tplug = structs.NewCSIPlugin(info.PluginID, index)\n\t\t}\n\n\t\t// the plugin may have been created by the job being updated, in which case\n\t\t// this data will not be configured, it's only available to the fingerprint\n\t\t// system","sourceCodeStart":1397,"sourceCodeEnd":1433,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1397-L1433","documentation":"Wraps a memdb lookup failure when fetching a CSIPlugin row by ID while registering a node's CSI plugins (upsertCSIPluginsForNode). The First() call on TableCSIPlugins returned an error, meaning the transaction or index is in a bad state — not merely that the plugin is missing (that yields raw == nil).","triggerScenarios":"Node registration (upsertNodeTxn) where the node's fingerprint includes CSIInfo entries and txn.First(TableCSIPlugins, \"id\", pluginID) errors, e.g. invalid index name or a failed transaction.","commonSituations":"Rarely hit in practice; appears in logs when a Nomad server's in-memory state is inconsistent, after a failed upgrade/migration, or in bug reports involving CSI plugin registration.","solutions":["Retry node registration — the error is typically transient state corruption during the transaction.","Check the wrapped error (%v suffix) for the actual memdb cause and address it specifically.","Confirm the client's CSI fingerprint data (plugin IDs) are well-formed; malformed IDs can trip index lookups.","Upgrade Nomad if on an older release with known CSI state-store bugs."],"exampleFix":"// before: plugin ID comes straight from fingerprint\ninfo.PluginID\n// after: validate before registration on the client side\nif info.PluginID == \"\" {\n\treturn fmt.Errorf(\"csi plugin fingerprint missing PluginID\")\n}","handlingStrategy":"retry","validationCode":"// client side: validate fingerprinted CSI plugins before registration\nfor _, info := range node.CSIControllerPlugins {\n    if info.PluginID == \"\" { return errors.New(\"CSI plugin fingerprint missing PluginID\") }\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"csi_plugin lookup error\") {\n    logger.Warn(\"csi plugin lookup failed during registration; retrying\", \"err\", err)\n    return retryNodeRegister(node)\n}","preventionTips":["Keep CSI plugin IDs stable and non-empty in your plugin task definitions.","Run current Nomad versions — early CSI state-store bugs were fixed.","Monitor server logs for repeated CSI errors on node registration.","Verify plugin jobs are healthy so fingerprint data stays consistent."],"tags":["nomad","csi","state-store","memdb"],"backgroundTag":"csi-plugin-registration-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}