hashicorp/nomad · error

failed to delete job %v (%d) from job_version

Error message

failed to delete job %v (%d) from job_version

What it means

This error occurs when the state store fails to delete the oldest tracked job version row from the job_version table during version trimming after a job update. The txn.Delete on the memdb table returned an error, so GC could not complete. It signals an internal persistence problem, since the row was just read in the same transaction.

Source

Thrown at nomad/state/state_store.go:2283

	stableIdx := -1
	for i, j := range all {
		if j.Stable {
			stableIdx = i
			break
		}
	}

	// If the stable job is the oldest version, do a swap to bring it into the
	// keep set.
	max := s.config.JobTrackedVersions
	if stableIdx == max {
		all[max-1], all[max] = all[max], all[max-1]
	}

	// Delete the oldest one
	d := all[max]
	if err := txn.Delete("job_version", d); err != nil {
		return fmt.Errorf("failed to delete job %v (%d) from job_version", d.ID, d.Version)
	}

	return nil
}

// GetJobSubmissions returns an iterator that contains all job submissions
// stored within state. This is not currently exposed via RPC and is only used
// for snapshot persist and restore functionality.
func (s *StateStore) GetJobSubmissions(ws memdb.WatchSet) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()

	// Walk the entire table to get all job submissions.
	iter, err := txn.Get(TableJobSubmission, indexID)
	if err != nil {
		return nil, fmt.Errorf("job submissions lookup failed: %v", err)
	}
	ws.Add(iter.WatchCh())

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the message's job ID and version to identify the affected row
2. Retry the job operation (txn is aborted atomically)
3. If reproducible, snapshot/restore Raft state and report to Nomad with the wrapped cause
4. Reduce job update churn so trimming happens less often

if err := register(); err != nil && strings.Contains(err.Error(), "from job_version") {
  time.Sleep(backoff)
  err = register() // txn aborted atomically, safe to retry
}

When it happens

Trigger: deleteJobVersion trimming the oldest version when len(job versions) > JobTrackedVersions and txn.Delete("job_version", ...) errors.

Common situations: Very rare: memdb write failures, transaction conflicts, or corruption; typically seen with internal state store bugs during frequent job updates.

Related errors

• deleting job versions failed: %v
• failed to insert job into job_version table: %v
• failed to look up job versions for %q: %v
• volume delete failed: %s: %v
• launch delete failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/66c4781e0b39c479. Report an issue: GitHub.