{"record":{"id":"32180d985fa4887a","repo":"hashicorp/nomad","slug":"csi-plugins-delete-error-v","errorCode":null,"errorMessage":"csi_plugins delete error: %v","messagePattern":"csi_plugins delete error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1573,"sourceCode":"\t\terr = updateOrGCPlugin(index, txn, plug)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tif err := txn.Insert(\"index\", &IndexEntry{TableCSIPlugins, index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// updateOrGCPlugin updates a plugin but will delete it if the plugin is empty\nfunc updateOrGCPlugin(index uint64, txn Txn, plug *structs.CSIPlugin) error {\n\tif plug.IsEmpty() {\n\t\terr := txn.Delete(TableCSIPlugins, plug)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"csi_plugins delete error: %v\", err)\n\t\t}\n\t} else {\n\t\tplug.ModifyIndex = index\n\t\terr := txn.Insert(TableCSIPlugins, plug)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"csi_plugins update error %s: %v\", plug.ID, err)\n\t\t}\n\t}\n\treturn nil\n}\n\n// deleteJobFromPlugins removes the allocations of this job from any plugins the job is\n// running, possibly deleting the plugin if it's no longer in use. It's called in DeleteJobTxn\nfunc (s *StateStore) deleteJobFromPlugins(index uint64, txn Txn, job *structs.Job) error {\n\tws := memdb.NewWatchSet()\n\tsummary, err := s.JobSummaryByID(ws, job.Namespace, job.ID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"error getting job summary: %v\", err)","sourceCodeStart":1555,"sourceCodeEnd":1591,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1555-L1591","documentation":"Wraps a memdb Delete failure when updateOrGCPlugin garbage-collects an empty CSIPlugin (no controllers, nodes, or allocations remain). The row cannot be deleted, failing the enclosing transaction (node upsert/delete, job delete, or alloc terminal update).","triggerScenarios":"Any caller (upsertCSIPluginsForNode, deleteNodeCSIPlugins, deleteJobFromPlugins, updatePluginForTerminalAlloc, updatePluginWithJobSummary) finds plug.IsEmpty() and txn.Delete(TableCSIPlugins, plug) errors.","commonSituations":"Hit when jobs using CSI volumes complete or nodes deregister and the now-unused plugin should be GC'd; indicates state-store inconsistency or a Nomad CSI bug.","solutions":["Retry the triggering operation (job deregistration, node delete, alloc update).","Check plugin state via nomad plugin status; force-deregister stale plugins with nomad plugin deregister if rows are stuck.","Inspect the wrapped %v error for the memdb root cause.","Upgrade Nomad; several CSI GC issues were fixed across patch releases."],"exampleFix":"// before: plugin left stuck, manual cleanup needed\n$ nomad plugin status\n$ nomad plugin deregister <plugin-id>\n// after: ensure CSI volumes/jobs referencing the plugin are purged first\n$ nomad volume deregister <volume-id>\n$ nomad job stop <csi-job>","handlingStrategy":"validation","validationCode":"// before GC-prone operations, ensure no references remain\nplugins, _, _ := client.CSIPlugins().List(nil)\nfor _, p := range plugins {\n    if p.ControllersHealthy == 0 && p.NodesHealthy == 0 {\n        client.CSIPlugins().Deregister(p.ID, nil) // explicit, controlled cleanup\n    }\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"csi_plugins delete error\") {\n    logger.Error(\"failed to GC empty CSI plugin; run nomad plugin deregister manually\", \"err\", err)\n    return manualPluginCleanup()\n}","preventionTips":["Deregister CSI volumes before stopping CSI controller plugin jobs.","Use nomad plugin status routinely to catch stuck plugins.","Use nomad plugin deregister for explicit cleanup instead of relying on implicit GC.","Upgrade Nomad — plugin GC behavior improved across releases."],"tags":["nomad","csi","state-store","garbage-collection"],"backgroundTag":"csi-plugin-gc-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}