What causes "Token does not match for Evaluation ID" in hashicorp/nomad?

Pass the exact token string returned by the matching Dequeue call.

Token does not match for Evaluation ID

Error message

Token does not match for Evaluation ID

What it means

Ack found the evaluation in the unacked map, but the supplied dequeue token does not match the token issued at Dequeue time. The token proves ownership of the in-flight evaluation; a mismatch means a different worker (or a stale worker) is trying to ack. This guards against double-processing across workers.

Source

Thrown at nomad/eval_broker.go:613

	return nil
}

// Ack is used to positively acknowledge handling an evaluation
func (b *EvalBroker) Ack(evalID, token string) error {
	b.l.Lock()
	defer b.l.Unlock()

	// Always delete the requeued evaluation. Either the Ack is successful and
	// we requeue it or it isn't and we want to remove it.
	defer delete(b.requeue, token)

	// Lookup the unack'd eval
	unack, ok := b.unack[evalID]
	if !ok {
		return fmt.Errorf("Evaluation ID not found")
	}
	if unack.Token != token {
		return fmt.Errorf("Token does not match for Evaluation ID")
	}
	jobID := unack.Eval.JobID

	defer b.handleAckNackLocked(unack.Eval)

	// Ensure we were able to stop the timer
	if !unack.NackTimer.Stop() {
		return fmt.Errorf("Evaluation ID Ack'd after Nack timer expiration")
	}

	// Update the stats
	b.stats.TotalUnacked -= 1
	queue := unack.Eval.Type
	if b.evals[evalID] > b.deliveryLimit {
		queue = failedQueue
	}
	bySched := b.stats.ByScheduler[queue]
	bySched.Unacked -= 1

View on GitHub (pinned to 482b49bf1a)

Solutions

Pass the exact token string returned by the matching Dequeue call.
Keep (evalID, token) paired in the worker's state; never cache tokens across restarts.
Fix variable shadowing/mix-ups where multiple dequeues are in flight.
If the token was lost, Nack cannot help either — let the nack timer expire and pick up the eval on a fresh Dequeue.

Example fix

// before
broker.Ack(eval.ID, oldToken) // token from earlier dequeue
// after
broker.Ack(eval.ID, dequeueToken) // token captured from same Dequeue result

Defensive patterns

Strategy: validation

Validate before calling

func validateAckPair(evalID, token string, issued map[string]string) error {
	if want, ok := issued[evalID]; !ok || want != token {
		return fmt.Errorf("token mismatch for eval %s", evalID)
	}
	return nil
}

Try / catch

if err := broker.Ack(evalID, token); err != nil {
	if strings.Contains(err.Error(), "Token does not match") {
		log.Printf("lost ownership of eval %s; skipping ack", evalID)
		return nil
	}
	return err
}

Prevention

Bind the token tightly to the dequeue in worker state (struct field, not global)
Never persist/reuse tokens across worker restarts
Guard against concurrent handlers sharing token variables
Use one goroutine per dequeue to avoid token mix-ups

When it happens

Trigger: Calling Ack(evalID, wrongToken): mixing up tokens between two dequeued evals, acking with a token from a previous dequeue of the same eval ID, or passing an empty/zero-value token.

Common situations: Worker crash and restart resends an old ack with a lost in-memory token; a shared handler closes over the wrong token variable; eval ID collisions in custom code that reuses eval IDs.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f124cf00aa3af88c. Report an issue: GitHub.