{"record":{"id":"c1e9db1d03718b21","repo":"hashicorp/nomad","slug":"csi-plugins-lookup-error-s-v","errorCode":null,"errorMessage":"csi_plugins lookup error %s: %v","messagePattern":"csi_plugins lookup error (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1542,"sourceCode":"\n// deleteNodeCSIPlugins cleans up CSIInfo node health status, called in DeleteNode\nfunc deleteNodeCSIPlugins(txn *txn, node *structs.Node, index uint64) error {\n\tif len(node.CSIControllerPlugins) == 0 && len(node.CSINodePlugins) == 0 {\n\t\treturn nil\n\t}\n\n\tnames := map[string]struct{}{}\n\tfor _, info := range node.CSIControllerPlugins {\n\t\tnames[info.PluginID] = struct{}{}\n\t}\n\tfor _, info := range node.CSINodePlugins {\n\t\tnames[info.PluginID] = struct{}{}\n\t}\n\n\tfor id := range names {\n\t\traw, err := txn.First(TableCSIPlugins, \"id\", id)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"csi_plugins lookup error %s: %v\", id, err)\n\t\t}\n\t\tif raw == nil {\n\t\t\t// plugin may have been deregistered but we didn't\n\t\t\t// update the fingerprint yet\n\t\t\tcontinue\n\t\t}\n\n\t\tplug := raw.(*structs.CSIPlugin).Copy()\n\t\terr = plug.DeleteNode(node.ID)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\terr = updateOrGCPlugin(index, txn, plug)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n","sourceCodeStart":1524,"sourceCodeEnd":1560,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1524-L1560","documentation":"Wraps a memdb First() lookup failure for a CSI plugin ID during deleteNodeCSIPlugins, which cleans plugin health when a node is deregistered (deleteNodeTxn). Note raw == nil (plugin absent) is handled gracefully with continue — this error only fires when the lookup itself errors.","triggerScenarios":"DeleteNode / node deregistration where the node carried CSIInfo; iterating the collected plugin IDs and txn.First(TableCSIPlugins, \"id\", id) returns a memdb error.","commonSituations":"Seen in server logs when draining or deregistering CSI-capable nodes; points to state-store inconsistency or a Nomad bug rather than operator action.","solutions":["Retry the node delete; the operation is idempotent via Raft replication.","Inspect the wrapped error's %v cause for the actual memdb failure.","If plugin rows are already inconsistent, use the CSI plugin API to check state and force-deregister stale plugins.","Upgrade Nomad if tied to a known CSI deregistration bug."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"csi_plugins lookup error\") {\n    logger.Warn(\"csi plugin lookup failed on node delete; retrying deregistration\", \"err\", err)\n    return retryNodeDeregister(nodeID)\n}","preventionTips":["Deregister CSI plugin jobs before removing nodes where possible.","Use nomad plugin status to audit stale plugins after node removal.","Force-deregister orphaned plugins with nomad plugin deregister.","Keep Nomad current; CSI node-drain bugs were fixed in patches."],"tags":["nomad","csi","state-store","memdb"],"backgroundTag":"csi-plugin-deregistration-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}