What causes "Evaluation ID not found" in hashicorp/nomad?

Ensure each Dequeue result is acked exactly once, within the nack timeout.

Evaluation ID not found

Error message

Evaluation ID not found

What it means

Ack is called by a scheduler worker to acknowledge a dequeued evaluation; this error means no evaluation with the given ID is currently in the unacked (in-flight) map b.unack. Either the eval was already acked/nacked, its nack timer expired and it was requeued, or the ID is simply wrong. Ack is idempotent-hostile by design: each dequeue must be acked exactly once.

Source

Thrown at nomad/eval_broker.go:610

	if !unack.NackTimer.Reset(b.nackTimeout) {
		return ErrNackTimeoutReached
	}
	return nil
}

// Ack is used to positively acknowledge handling an evaluation
func (b *EvalBroker) Ack(evalID, token string) error {
	b.l.Lock()
	defer b.l.Unlock()

	// Always delete the requeued evaluation. Either the Ack is successful and
	// we requeue it or it isn't and we want to remove it.
	defer delete(b.requeue, token)

	// Lookup the unack'd eval
	unack, ok := b.unack[evalID]
	if !ok {
		return fmt.Errorf("Evaluation ID not found")
	}
	if unack.Token != token {
		return fmt.Errorf("Token does not match for Evaluation ID")
	}
	jobID := unack.Eval.JobID

	defer b.handleAckNackLocked(unack.Eval)

	// Ensure we were able to stop the timer
	if !unack.NackTimer.Stop() {
		return fmt.Errorf("Evaluation ID Ack'd after Nack timer expiration")
	}

	// Update the stats
	b.stats.TotalUnacked -= 1
	queue := unack.Eval.Type
	if b.evals[evalID] > b.deliveryLimit {
		queue = failedQueue

View on GitHub (pinned to 482b49bf1a)

Solutions

Ensure each Dequeue result is acked exactly once, within the nack timeout.
Check that the caller stored the correct evalID/token pair returned from Dequeue.
Increase the nack timeout if handlers legitimately take longer than the current timeout.
Treat this error as benign in retry logic — the eval was requeued and will be redelivered; just drop the stale ack.
Verify the handler isn't acking after a Nack for the same eval.

Example fix

// before
func handle(ev *structs.Evaluation, tok string) {
	process(ev) // may exceed nack timeout
	broker.Ack(ev.ID, tok) // may hit "not found"
}
// after
done := make(chan struct{})
go func() { process(ev); close(done) }()
select {
case <-done:
	broker.Ack(ev.ID, tok)
case <-time.After(nackTimeout/2):
	broker.Nack(ev.ID, tok) // explicitly give up before timer fires
}

Defensive patterns

Strategy: try-catch

Validate before calling

// track in-flight (evalID -> token) pairs per worker
inflight := map[string]string{}
func canAck(evalID, token string) bool {
	t, ok := inflight[evalID]
	return ok && t == token
}

Try / catch

if err := broker.Ack(evalID, token); err != nil {
	if strings.Contains(err.Error(), "Evaluation ID not found") {
		// already acked/nacked or timer fired; eval was requeued
		log.Printf("stale ack for eval %s", evalID)
		return nil
	}
	return err
}

Prevention

Ack within the nack timeout, never after
Ack exactly once per dequeue
Store the Dequeue (evalID, token) pair together
Keep handler latency well below the nack timeout

When it happens

Trigger: Calling Ack(evalID, token) with an evalID that was never dequeued by this broker, calling it twice for the same dequeue, or calling it after the nack timer already fired and removed the unack entry.

Common situations: Slow worker acks after the nack timeout elapsed so the eval was requeued to another worker; duplicate Ack from a retried handler; a bug where evalID/token get swapped or truncated; long-blocking handler exceeding nackTimeout.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/bb4f5921d94f770e. Report an issue: GitHub.