hashicorp/nomad · error

setting job status failed: %v

Error message

setting job status failed: %v

What it means

Returned by StateStore.UpsertEvals (after eval insert) when the follow-up s.setJobStatuses call fails while recomputing and setting the statuses of jobs affected by the evaluations. The evals themselves may be staged, but because job status update failed the transaction is abandoned and this wrapper error is what callers see.

Source

Thrown at nomad/state/state_store.go:3469

// in a transaction.  Useful for when making multiple modifications atomically.
func (s *StateStore) UpsertEvalsTxn(index uint64, evals []*structs.Evaluation, txn Txn) error {
	// Do a nested upsert
	jobs := make(map[structs.NamespacedID]string, len(evals))
	for _, eval := range evals {
		if err := s.nestedUpsertEval(txn, index, eval); err != nil {
			return err
		}

		tuple := structs.NamespacedID{
			ID:        eval.JobID,
			Namespace: eval.Namespace,
		}
		jobs[tuple] = ""
	}

	// Set the job's status
	if err := s.setJobStatuses(index, txn, jobs, false); err != nil {
		return fmt.Errorf("setting job status failed: %v", err)
	}

	return nil
}

// nestedUpsertEvaluation is used to nest an evaluation upsert within a transaction
func (s *StateStore) nestedUpsertEval(txn *txn, index uint64, eval *structs.Evaluation) error {
	// Lookup the evaluation
	existing, err := txn.First("evals", "id", eval.ID)
	if err != nil {
		return fmt.Errorf("eval lookup failed: %v", err)
	}

	// Update the indexes
	if existing != nil {
		eval.CreateIndex = existing.(*structs.Evaluation).CreateIndex
		eval.ModifyIndex = index
	} else {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped inner error from setJobStatuses in the message
2. Verify the evals reference existing jobs (namespace + job ID) in state
3. Retry the operation; if evals for deleted jobs keep arriving, investigate the scheduler/eval producer
4. Restart servers if the inner error indicates memdb corruption

// Ensure every eval references a live job before UpsertEvals
for _, e := range evals {
	job, err := state.JobByID(ws, e.Namespace, e.JobID)
	if err != nil { return err }
	if job == nil { /* drop or re-create eval */ }
}

if ev, ok := raw.(*structs.Evaluation); ok { /* use ev */ }

if err := state.UpsertEvals(idx, evals); err != nil {
	if strings.Contains(err.Error(), "setting job status failed") {
		// inspect inner cause: missing job vs memdb write failure
	}
	return err
}

When it happens

Trigger: Calling UpsertEvals where setJobStatuses fails internally — e.g. it cannot look up the job(s) referenced by the evals, or the internal job-status insert/update into memdb errors (job missing from state, or memdb write failure).

Common situations: Evals referencing jobs that were deregistered concurrently (race between eval creation and job delete); jobs in a different namespace than expected; state store write failures under memory pressure.

Related errors

• eval broker is enabled; eval broker must be paused to delete
• job summary lookup failed: %v
• eval delete failed: %v
• failed to lookup job for eval: %v
• failed to lookup eval allocs: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/230c9b785abaf790. Report an issue: GitHub.