hashicorp/nomad · error

UpsertJob failed: %w

Error message

UpsertJob failed: %w

What it means

After copying the current job and marking it stopped, handleJobDeregister re-inserts it via state.UpsertJobTxn within the same transaction. Failure there (schema/validation/constraint issues in the state store) is wrapped as "UpsertJob failed" and aborts the deregistration apply.

Source

Thrown at nomad/fsm.go:933

	if current == nil {
		return fmt.Errorf("job %q in namespace %q doesn't exist to be deregistered", jobID, namespace)
	}

	stopped := current.Copy()
	stopped.Stop = true
	if submitTime != 0 {
		stopped.SubmitTime = submitTime
	}

	// Disable scaling policies to avoid monitoring stopped jobs
	scalingPolicies := stopped.GetScalingPolicies()
	for _, policy := range scalingPolicies {
		policy.Enabled = false
	}

	if err := n.state.UpsertJobTxn(index, nil, stopped, tx); err != nil {
		return fmt.Errorf("UpsertJob failed: %w", err)
	}

	return nil
}

func (n *nomadFSM) applyUpdateEval(msgType structs.MessageType, buf []byte, index uint64) any {
	defer metrics.MeasureSince([]string{"nomad", "fsm", "update_eval"}, time.Now())

	var req structs.EvalUpdateRequest
	if err := structs.Decode(buf, &req); err != nil {
		panic(fmt.Errorf("failed to decode request: %v", err))
	}

	return n.upsertEvals(msgType, index, req.Evals)
}

func (n *nomadFSM) upsertEvals(msgType structs.MessageType, index uint64, evals []*structs.Evaluation) error {
	if err := n.state.UpsertEvals(msgType, index, evals); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped underlying error for the exact state-store rejection reason.
2. Ensure all servers run a compatible Nomad version (`nomad server members` shows versions); upgrade stragglers.
3. Restart the server to rebuild in-memory state from Raft snapshots.
4. Restore a consistent snapshot if state corruption is suspected.

err := client.Jobs().Deregister(jobID, false, nil)
if err != nil && strings.Contains(err.Error(), "UpsertJob failed") {
    // inspect wrapped cause; upgrade mismatched servers and retry
}

When it happens

Trigger: UpsertJobTxn rejects the stopped job copy — e.g. job spec validation within the state store fails, index conflict, or memdb write error during FSM apply.

Common situations: Jobs written by newer Nomad versions then applied by an older server (schema drift), corrupted state, or transaction-level errors after earlier steps.

Related errors

• index update failed: %v
• JobByID lookup failed: %w
• error querying plugin %q: %v
• failed adding job to periodic dispatcher: %v
• failed to retrieve latest deployment: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ecbc799703a5db37. Report an issue: GitHub.