hashicorp/nomad · error

failed to determine last evaluation index for job %q: %v

Error message

failed to determine last evaluation index for job %q: %v

What it means

Inside the deployment watcher, when an allocation update arrives, the watcher needs the job's latest evaluation to know the current deployment state. If jobEvalStatus (a state-store read) fails, the alloc update is dropped and this wrapped error is returned; the watcher loop then retries via the watch loop.

Source

Thrown at nomad/deploymentwatcher/deployment_watcher.go:608

	createEval        bool
	failDeployment    bool
	rollback          bool
	allowReplacements []string
}

// handleAllocUpdate is used to compute the set of actions to take based on the
// updated allocations for the deployment.
func (w *deploymentWatcher) handleAllocUpdate(allocs []*structs.AllocListStub) (allocUpdateResult, error) {
	var res allocUpdateResult

	// Get the latest evaluation index
	latestEval, err := w.jobEvalStatus()
	if err != nil {
		if err == context.Canceled || w.ctx.Err() == context.Canceled {
			return res, err
		}

		return res, fmt.Errorf("failed to determine last evaluation index for job %q: %v", w.j.ID, err)
	}

	deployment := w.getDeployment()
	for _, alloc := range allocs {
		dstate, ok := deployment.TaskGroups[alloc.TaskGroup]
		if !ok {
			continue
		}

		// Check if we can already fail the deployment
		failDeployment := w.shouldFailEarly(deployment, alloc, dstate)

		// Check if the allocation has failed and we need to mark it for allow
		// replacements
		if alloc.DeploymentStatus.IsUnhealthy() && !failDeployment &&
			deployment.Active() && !alloc.DesiredTransition.ShouldReschedule() {
			res.allowReplacements = append(res.allowReplacements, alloc.ID)
			continue

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check server logs around the error for the underlying state-store/Raft cause
2. If it followed a leader election, it is usually transient — the watcher re-runs; verify the deployment progresses with `nomad deployment status`
3. If persistent, inspect `nomad server members` for Raft health and restart/replace the failing server

// caller-side: monitor server health before/while deployments run
nomad server members   # ensure the leader is stable and Raft is healthy
nomad deployment status <id>  # confirm progress despite transient watcher errors

// server-side watcher already retries via its watch loop; client-side, poll until settled:
for i := 0; i < 10; i++ {
    d, _, err := client.Deployments().Info(deployID, nil)
    if err == nil && (d.Status == "successful" || d.Status == "failed" || d.Status == "cancelled") { return d, nil }
    time.Sleep(2 * time.Second)
}
return nil, errors.New("deployment did not settle")

When it happens

Trigger: State store read errors for the job's evaluations while processing alloc updates — typically during Raft instability, FSM/state-store restore, or a context cancellation racing the lookup (cancellation is specially handled and passes through).

Common situations: Leader failover mid-deployment; server under heavy load with state store contention; cluster restore where the job was GC'd between deployment creation and alloc updates.

Related errors

• error getting plugin: %s, %v
• failed to retrieve jobs for idempotency check
• failed to snapshot state: %v
• failed to find allocs for '%s': %v
• failed to find system jobs for '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/99837821d3d563b8. Report an issue: GitHub.