hashicorp/nomad · error

failed to insert job into job_version table: %v

Error message

failed to insert job into job_version table: %v

What it means

upsertJobVersion inserts the submitted job snapshot into the "job_version" table to record job history. This wrapped error means the row insert failed inside memdb, so the new job version is not recorded and the registration transaction aborts and rolls back.

Source

Thrown at nomad/state/state_store.go:2244

	if err := txn.Insert("index", &IndexEntry{"job_version", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// upsertJobVersion inserts a job into its historic version table and limits the
// number of job versions that are tracked.
func (s *StateStore) upsertJobVersion(index uint64, job *structs.Job, txn *txn) error {
	// JobTrackedVersions really must not be zero here
	if err := s.config.Validate(); err != nil {
		return err
	}

	// Insert the job
	if err := txn.Insert("job_version", job); err != nil {
		return fmt.Errorf("failed to insert job into job_version table: %v", err)
	}

	if err := txn.Insert("index", &IndexEntry{"job_version", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	// Get all the historic jobs for this ID, except those with a VersionTag,
	// as they should always be kept. They are in Version order, high to low.
	all, err := s.jobVersionByID(txn, nil, job.Namespace, job.ID, false)
	if err != nil {
		return fmt.Errorf("failed to look up job versions for %q: %v", job.ID, err)
	}

	// If we are below the limit there is no GCing to be done
	if len(all) <= s.config.JobTrackedVersions {
		return nil
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped cause — a duplicate-key error indicates the Version already exists, so submit with a unique/incremented version.
2. Retry the registration on a fresh transaction if the cause is transient.
3. Ensure restored/migrated state does not reuse existing (Namespace, ID, Version) keys.
4. Verify job struct validity (non-empty namespace/ID/version) before upserting.

Example fix

// before (caller re-submitting a job with an existing Version)
job.Version = 3 // already stored
err := state.UpsertJob(msgType, index, job)
// after
job.Version = existingVersion + 1 // unique
err := state.UpsertJob(msgType, index, job)

// validate job identity and version uniqueness before upsert
if job == nil || job.ID == "" || job.Namespace == "" {
	return fmt.Errorf("job must have namespace and ID")
}
existing, _ := state.jobVersionByID(nil, nil, job.Namespace, job.ID, false)
for _, j := range existing {
	if j.Version == job.Version {
		return fmt.Errorf("version %d already exists for %s/%s", job.Version, job.Namespace, job.ID)
	}
}

func jobVersionIsUnique(existing []*structs.Job, version uint64) bool {
	for _, j := range existing {
		if j.Version == version { return false }
	}
	return true
}

err := state.UpsertJob(msgType, index, job)
if err != nil {
	if strings.Contains(err.Error(), "job_version table") {
		// possible duplicate key: resubmit with next version
		job.Version++
		return state.UpsertJob(msgType, index, job)
	}
	return err
}

When it happens

Trigger: Job registration/update when txn.Insert("job_version", job) errors — memdb allocation failure, already-aborted transaction, a duplicate primary key conflict (same Namespace/JobID/Version already present), or passing an invalid job struct.

Common situations: Job submissions with a Version colliding with an existing historical row (custom tooling or restores that mishandled version numbers); state store churn under load; snapshot/restore windows.

Related errors

• deleting job versions failed: %v
• failed to look up job versions for %q: %v
• failed to delete job %v (%d) from job_version
• tag %q already exists on a different version of job %q
• job %q version %d not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5c5ea38e89698d21. Report an issue: GitHub.