hashicorp/nomad · error

deleting job versions failed: %v

Error message

deleting job versions failed: %v

What it means

deleteJobByVersion (historic job version cleanup) deletes collected old job versions one by one with txn.Delete on the "job_version" table. This wrapped error means one of those deletions failed inside memdb, aborting the version-pruning transaction and preventing the new job version from being persisted cleanly.

Source

Thrown at nomad/state/state_store.go:2223

	for {
		raw := iter.Next()
		if raw == nil {
			break
		}

		// Ensure the ID is an exact match
		j := raw.(*structs.Job)
		if j.ID != job.ID {
			continue
		}

		jobs = append(jobs, j)
	}

	// Do the deletes
	for _, j := range jobs {
		if err := txn.Delete("job_version", j); err != nil {
			return fmt.Errorf("deleting job versions failed: %v", err)
		}
	}

	if err := txn.Insert("index", &IndexEntry{"job_version", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// upsertJobVersion inserts a job into its historic version table and limits the
// number of job versions that are tracked.
func (s *StateStore) upsertJobVersion(index uint64, job *structs.Job, txn *txn) error {
	// JobTrackedVersions really must not be zero here
	if err := s.config.Validate(); err != nil {
		return err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause for the specific failing version row.
2. Retry the job registration; version collection and deletion are recomputed on a fresh txn.
3. Avoid concurrent submissions of the same job (Nomad handles this via Raft serialization).
4. Check job_version table integrity if errors repeat.

Example fix

// before
if err := txn.Delete("job_version", j); err != nil {
	return fmt.Errorf("deleting job versions failed: %v", err)
}
// after
if err := txn.Delete("job_version", j); err != nil {
	return fmt.Errorf("deleting job versions failed: %w", err)
}

// check existing versions before re-registering a job
versions, err := state.jobVersionByID(nil, nil, ns, jobID, false)
if err != nil {
	return err
}
maxVersion := uint64(0)
for _, v := range versions {
	if v.Version > maxVersion { maxVersion = v.Version }
}
_ = maxVersion // submit new job with maxVersion+1

err := state.UpsertJob(msgType, index, job)
if err != nil && strings.Contains(err.Error(), "deleting job versions") {
	time.Sleep(backoff)
	return state.UpsertJob(msgType, index, job)
}

When it happens

Trigger: Job registration/update (upsertJobVersion pruning history) when txn.Delete("job_version", j) errors for a collected old version — aborted transaction, memdb internal error, or a row deleted concurrently.

Common situations: Very frequent job updates keeping many historical versions (pruning triggers on every registration); concurrent registrations of the same job; state churn under load.

Related errors

• failed to insert job into job_version table: %v
• failed to look up job versions for %q: %v
• failed to delete job %v (%d) from job_version
• tag %q already exists on a different version of job %q
• job %q version %d not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6f85b5f95f879947. Report an issue: GitHub.