{"record":{"id":"9cf824bd35525b89","repo":"hashicorp/nomad","slug":"csi-plugins-update-error-s-v","errorCode":null,"errorMessage":"csi_plugins update error %s: %v","messagePattern":"csi_plugins update error (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1579,"sourceCode":"\tif err := txn.Insert(\"index\", &IndexEntry{TableCSIPlugins, index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// updateOrGCPlugin updates a plugin but will delete it if the plugin is empty\nfunc updateOrGCPlugin(index uint64, txn Txn, plug *structs.CSIPlugin) error {\n\tif plug.IsEmpty() {\n\t\terr := txn.Delete(TableCSIPlugins, plug)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"csi_plugins delete error: %v\", err)\n\t\t}\n\t} else {\n\t\tplug.ModifyIndex = index\n\t\terr := txn.Insert(TableCSIPlugins, plug)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"csi_plugins update error %s: %v\", plug.ID, err)\n\t\t}\n\t}\n\treturn nil\n}\n\n// deleteJobFromPlugins removes the allocations of this job from any plugins the job is\n// running, possibly deleting the plugin if it's no longer in use. It's called in DeleteJobTxn\nfunc (s *StateStore) deleteJobFromPlugins(index uint64, txn Txn, job *structs.Job) error {\n\tws := memdb.NewWatchSet()\n\tsummary, err := s.JobSummaryByID(ws, job.Namespace, job.ID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"error getting job summary: %v\", err)\n\t}\n\n\tallocs, err := s.AllocsByJob(ws, job.Namespace, job.ID, false)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"error getting allocations: %v\", err)\n\t}","sourceCodeStart":1561,"sourceCodeEnd":1597,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1561-L1597","documentation":"Wraps a memdb Insert failure when updateOrGCPlugin re-persists a still-in-use CSIPlugin with a new ModifyIndex. Unlike 2527 the plugin is not empty, so it is updated rather than deleted; the insert failed and fails the enclosing transaction.","triggerScenarios":"Callers updatePluginForTerminalAlloc / updatePluginWithJobSummary / deleteJobFromPlugins / node upsert-delete paths reaching the else branch and txn.Insert(TableCSIPlugins, plug) erroring.","commonSituations":"Seen during alloc completion or job teardown for CSI workloads; generally an internal state-store failure rather than operator error.","solutions":["Retry the triggering operation.","Inspect the wrapped plug.ID and %v error to identify the failing plugin and memdb cause.","Verify healthy Raft/state-store operation on the server leader; restart if degraded.","Upgrade Nomad if reproducible on your release."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"csi_plugins update error\") {\n    pluginID := extractQuotedID(err) // parse %s from message if needed\n    logger.Warn(\"csi plugin update failed\", \"plugin\", pluginID, \"err\", err)\n    return backoffRetry(triggeringOp)\n}","preventionTips":["Keep alloc churn on CSI jobs moderate; batch terminal updates where possible.","Monitor server CPU/memory during mass job stops.","Keep Nomad patched for CSI alloc-update fixes.","Verify plugin health before scaling CSI workloads."],"tags":["nomad","csi","state-store","memdb"],"backgroundTag":"csi-plugin-registration-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}