hashicorp/nomad · error

unable to update job csi plugins: %v

Error message

unable to update job csi plugins: %v

What it means

UpsertJob failed while reconciling job CSI plugins via updateJobCSIPlugins, which maintains the CSI plugin table entries when a job declares CSI plugin stanzas. The wrapped error aborts the entire registration transaction.

Source

Thrown at nomad/state/state_store.go:1900

	if err := s.updateSummaryWithJob(index, job, txn); err != nil {
		return fmt.Errorf("unable to create job summary: %v", err)
	}

	if err := s.upsertJobVersion(index, job, txn); err != nil {
		return fmt.Errorf("unable to upsert job into job_version table: %v", err)
	}

	if err := s.updateJobScalingPolicies(index, job, txn); err != nil {
		return fmt.Errorf("unable to update job scaling policies: %v", err)
	}

	if err := s.updateJobRecommendations(index, txn, existingJob, job); err != nil {
		return fmt.Errorf("unable to update job recommendations: %v", err)
	}

	if err := s.updateJobCSIPlugins(index, job, existingJob, txn); err != nil {
		return fmt.Errorf("unable to update job csi plugins: %v", err)
	}

	if err := s.updateJobSubmission(index, sub, job.Namespace, job.ID, job.Version, txn); err != nil {
		return fmt.Errorf("unable to update job submission: %v", err)
	}

	if err := s.updatePreservedValues(job, existingJob, req); err != nil {
		return fmt.Errorf("unable to update preserved values: %v", err)
	}

	// Insert the job
	if err := txn.Insert("jobs", job); err != nil {
		return fmt.Errorf("job insert failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"jobs", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause in server logs
2. Ensure the new CSI plugin job spec keeps the same plugin ID and does not conflict with existing plugin entries
3. Deregister dependent volumes/plugins (nomad volume deregister, nomad plugin deregister) before changing plugin jobs
4. Retry registration; the transaction rolled back atomically

// Verify CSI plugin spec consistency before re-registering
for _, tg := range job.TaskGroups {
  for _, v := range tg.Volumes {
    _ = v.Source // ensure volumes referencing the plugin are deregistered first
  }
  if tg.CSIPlugin != nil && oldJob != nil && oldPluginID != *tg.CSIPluginPluginID() {
    return fmt.Errorf("changing plugin ID for %s requires plugin deregistration", tg.Name)
  }
}

func isCSIPluginError(err error) bool {
  return err != nil && strings.Contains(err.Error(), "job csi plugins")
}

_, err := client.Jobs().Register(job, nil, nil, nil)
if isCSIPluginError(err) {
  // deregister dependent volumes/plugins, then retry
  _, derr := client.CSIVolumes().Deregister(volumeID, true, nil)
  if derr == nil {
    _, err = client.Jobs().Register(job, nil, nil, nil)
  }
}
return err

When it happens

Trigger: Submitting or updating a job with a csi_plugin block where plugin registration/deregistration in the state store fails (mismatch between old and new plugin job specs).

Common situations: Re-registering CSI plugin jobs with changed plugin IDs or controllers/nodes counts, or removing a CSI plugin job while volumes still reference it.

Related errors

• error parsing: root should be an object
• cannot specify Accessor ID
• failed to read dynamic plugin registry state: %v
• error getting plugin: %s, %v
• error querying volume %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/56205ba5bcea428d. Report an issue: GitHub.