hashicorp/nomad · error

failed to look up job versions for %q: %v

Error message

failed to look up job versions for %q: %v

What it means

This error is wrapped inside the job-version garbage collection logic in Nomad's state store. When the number of historic versions of a job exceeds JobTrackedVersions, the store GCs the oldest versions; if reading all versions of the job ID via jobVersionByID fails, this error is returned wrapping the underlying cause. It means the internal job_version table could not be iterated, typically a memdb/index-level failure.

Source

Thrown at nomad/state/state_store.go:2255

	// JobTrackedVersions really must not be zero here
	if err := s.config.Validate(); err != nil {
		return err
	}

	// Insert the job
	if err := txn.Insert("job_version", job); err != nil {
		return fmt.Errorf("failed to insert job into job_version table: %v", err)
	}

	if err := txn.Insert("index", &IndexEntry{"job_version", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	// Get all the historic jobs for this ID, except those with a VersionTag,
	// as they should always be kept. They are in Version order, high to low.
	all, err := s.jobVersionByID(txn, nil, job.Namespace, job.ID, false)
	if err != nil {
		return fmt.Errorf("failed to look up job versions for %q: %v", job.ID, err)
	}

	// If we are below the limit there is no GCing to be done
	if len(all) <= s.config.JobTrackedVersions {
		return nil
	}

	// We have to delete a historic job to make room.
	// Find index of the highest versioned stable job
	stableIdx := -1
	for i, j := range all {
		if j.Stable {
			stableIdx = i
			break
		}
	}

	// If the stable job is the oldest version, do a swap to bring it into the

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v cause for the underlying memdb/txn error
2. Retry the job registration; state store ops are transactional and transient errors rarely persist
3. If persistent, capture a debug bundle and file with Nomad; may indicate state corruption
4. Check JobTrackedVersions config and job churn; reduce version churn to avoid trimming paths

// pre-check version count via API
status, _ := client.Jobs().Info(jobID)
_ = status // ensure job exists and churn is low

// retry transient state store errors
for i := 0; i < 3; i++ {
  _, err := client.Jobs().Register(job, nil)
  if err == nil || !strings.Contains(err.Error(), "failed to look up job versions") { break }
  time.Sleep(backoff)
}

When it happens

Trigger: Calling job registration/delete paths that trigger version trimming (deleteJobVersion) when jobVersionByID fails to look up versions for the job ID in the given namespace, e.g. corrupted txn index or internal table read error.

Common situations: Rare internal state-store failures: memdb index errors, transaction invalidation, or bugs during job updates on clusters with many job versions (JobTrackedVersions exceeded).

Related errors

• deleting job versions failed: %v
• failed to insert job into job_version table: %v
• failed to delete job %v (%d) from job_version
• tag %q already exists on a different version of job %q
• job %q version %d not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/861a52f161697e53. Report an issue: GitHub.