What causes "upserting evals failed: %v" in hashicorp/nomad?

Inspect the wrapped UpsertEvalsTxn error cause (often the per-eval index update).

upserting evals failed: %v

Error message

upserting evals failed: %v

What it means

Thrown during client alloc update processing when s.UpsertEvalsTxn fails to write the evaluations accompanying the alloc update. The evals are upserted in the same transaction as the allocs, so any eval write error rolls back the entire update batch.

Source

Thrown at nomad/state/state_store.go:4063

	populatedAllocs := []*structs.Allocation{}
	// Handle each of the updated allocations
	for _, a := range allocs {
		nodeIDs.Insert(a.NodeID)
		ca, err := s.nestedUpdateAllocFromClient(txn, index, a)
		if ca == nil {
			continue
		}

		if err != nil {
			return fmt.Errorf("updating alloc failed: %v", err)
		}
		populatedAllocs = append(populatedAllocs, ca)
	}

	if len(req.Evals) > 0 {
		err := s.UpsertEvalsTxn(index, evals, txn)
		if err != nil {
			return fmt.Errorf("upserting evals failed: %v", err)
		}
	}

	jobs := map[structs.NamespacedID]string{}

	for _, alloc := range populatedAllocs {
		tuple := structs.NamespacedID{
			ID:        alloc.JobID,
			Namespace: alloc.Namespace,
		}

		jobs[tuple] = ""
	}

	if err := s.setJobStatuses(index, txn, jobs, false); err != nil {
		return fmt.Errorf("setting job status failed: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped UpsertEvalsTxn error cause (often the per-eval index update).
Check server logs and Raft apply health.
Restart the Nomad server if the store is wedged.
Clients re-drive evals, so no manual resubmission is usually needed once the server is healthy.

Defensive patterns

Strategy: retry

Try / catch

if strings.Contains(err.Error(), "upserting evals failed") {
  // whole txn rolled back; safe to retry once server recovers
  retryWithBackoff(func() error { return upsertAllocsAndEvals(req) })
}

Prevention

Rely on transactional rollback: the alloc batch is atomic, retry the whole batch.
Keep eval volumes within Nomad's tested limits per update.
Monitor Raft apply latency.
Upgrade Nomad if the underlying UpsertEvalsTxn cause matches a known bug.

When it happens

Trigger: Node.UpdateAllocs or UpdateEval requests carrying req.Evals where UpsertEvalsTxn hits a memdb write/index error for the evals table.

Common situations: Server-side failures while persisting follow-up evals created by alloc updates; seen in server logs during state store contention or corruption.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e21847cd5384fa8a. Report an issue: GitHub.