{"record":{"id":"56205ba5bcea428d","repo":"hashicorp/nomad","slug":"unable-to-update-job-csi-plugins-v","errorCode":null,"errorMessage":"unable to update job csi plugins: %v","messagePattern":"unable to update job csi plugins: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1900,"sourceCode":"\n\tif err := s.updateSummaryWithJob(index, job, txn); err != nil {\n\t\treturn fmt.Errorf(\"unable to create job summary: %v\", err)\n\t}\n\n\tif err := s.upsertJobVersion(index, job, txn); err != nil {\n\t\treturn fmt.Errorf(\"unable to upsert job into job_version table: %v\", err)\n\t}\n\n\tif err := s.updateJobScalingPolicies(index, job, txn); err != nil {\n\t\treturn fmt.Errorf(\"unable to update job scaling policies: %v\", err)\n\t}\n\n\tif err := s.updateJobRecommendations(index, txn, existingJob, job); err != nil {\n\t\treturn fmt.Errorf(\"unable to update job recommendations: %v\", err)\n\t}\n\n\tif err := s.updateJobCSIPlugins(index, job, existingJob, txn); err != nil {\n\t\treturn fmt.Errorf(\"unable to update job csi plugins: %v\", err)\n\t}\n\n\tif err := s.updateJobSubmission(index, sub, job.Namespace, job.ID, job.Version, txn); err != nil {\n\t\treturn fmt.Errorf(\"unable to update job submission: %v\", err)\n\t}\n\n\tif err := s.updatePreservedValues(job, existingJob, req); err != nil {\n\t\treturn fmt.Errorf(\"unable to update preserved values: %v\", err)\n\t}\n\n\t// Insert the job\n\tif err := txn.Insert(\"jobs\", job); err != nil {\n\t\treturn fmt.Errorf(\"job insert failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"jobs\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n","sourceCodeStart":1882,"sourceCodeEnd":1918,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1882-L1918","documentation":"UpsertJob failed while reconciling job CSI plugins via updateJobCSIPlugins, which maintains the CSI plugin table entries when a job declares CSI plugin stanzas. The wrapped error aborts the entire registration transaction.","triggerScenarios":"Submitting or updating a job with a csi_plugin block where plugin registration/deregistration in the state store fails (mismatch between old and new plugin job specs).","commonSituations":"Re-registering CSI plugin jobs with changed plugin IDs or controllers/nodes counts, or removing a CSI plugin job while volumes still reference it.","solutions":["Inspect the wrapped cause in server logs","Ensure the new CSI plugin job spec keeps the same plugin ID and does not conflict with existing plugin entries","Deregister dependent volumes/plugins (nomad volume deregister, nomad plugin deregister) before changing plugin jobs","Retry registration; the transaction rolled back atomically"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify CSI plugin spec consistency before re-registering\nfor _, tg := range job.TaskGroups {\n  for _, v := range tg.Volumes {\n    _ = v.Source // ensure volumes referencing the plugin are deregistered first\n  }\n  if tg.CSIPlugin != nil && oldJob != nil && oldPluginID != *tg.CSIPluginPluginID() {\n    return fmt.Errorf(\"changing plugin ID for %s requires plugin deregistration\", tg.Name)\n  }\n}","typeGuard":"func isCSIPluginError(err error) bool {\n  return err != nil && strings.Contains(err.Error(), \"job csi plugins\")\n}","tryCatchPattern":"_, err := client.Jobs().Register(job, nil, nil, nil)\nif isCSIPluginError(err) {\n  // deregister dependent volumes/plugins, then retry\n  _, derr := client.CSIVolumes().Deregister(volumeID, true, nil)\n  if derr == nil {\n    _, err = client.Jobs().Register(job, nil, nil, nil)\n  }\n}\nreturn err","preventionTips":["Never change a CSI plugin job's plugin ID between versions","Deregister volumes before removing a CSI plugin job","Keep controller/node counts within supported deltas when updating","Test CSI job updates in non-production first"],"tags":["nomad","state-store","csi"],"backgroundTag":"csi-plugin-sync-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}