hashicorp/nomad · error

Evaluation ID Ack'd after Nack timer expiration

Error message

Evaluation ID Ack'd after Nack timer expiration

What it means

Ack located the unacked eval and the token matched, but NackTimer.Stop() returned false, meaning the nack timer had already fired. The broker has already treated the eval as nacked (it was requeued/re-delivered), so this ack arrives too late and is rejected. This prevents the same evaluation from being acknowledged after it was given to another worker.

Source

Thrown at nomad/eval_broker.go:621

	// Always delete the requeued evaluation. Either the Ack is successful and
	// we requeue it or it isn't and we want to remove it.
	defer delete(b.requeue, token)

	// Lookup the unack'd eval
	unack, ok := b.unack[evalID]
	if !ok {
		return fmt.Errorf("Evaluation ID not found")
	}
	if unack.Token != token {
		return fmt.Errorf("Token does not match for Evaluation ID")
	}
	jobID := unack.Eval.JobID

	defer b.handleAckNackLocked(unack.Eval)

	// Ensure we were able to stop the timer
	if !unack.NackTimer.Stop() {
		return fmt.Errorf("Evaluation ID Ack'd after Nack timer expiration")
	}

	// Update the stats
	b.stats.TotalUnacked -= 1
	queue := unack.Eval.Type
	if b.evals[evalID] > b.deliveryLimit {
		queue = failedQueue
	}
	bySched := b.stats.ByScheduler[queue]
	bySched.Unacked -= 1

	// Cleanup
	delete(b.unack, evalID)
	delete(b.evals, evalID)

	namespacedID := structs.NamespacedID{
		ID:        jobID,
		Namespace: unack.Eval.Namespace,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Increase the eval broker nack timeout (NewEvalBroker timeout / server config) to exceed worst-case handler latency.
2. Make handlers faster or process asynchronously with the ack issued promptly after dequeue of the result.
3. Treat this error as benign: the eval was already requeued; discard the stale ack and any duplicated work should be idempotent.
4. Instrument handler duration vs nackTimeout to detect chronic overruns.
5. Avoid blocking the worker between Dequeue and Ack with synchronous long-running RPCs.

Example fix

// before
processLongJob(eval) // 5 min
broker.Ack(eval.ID, token) // nack timer (1 min) already fired
// after
ackCh := make(chan struct{})
go func() { processLongJob(eval); close(ackCh) }()
select {
case <-ackCh:
	broker.Ack(eval.ID, token)
case <-time.After(30 * time.Second):
	broker.Nack(eval.ID, token)
}

// pre-check: handler deadline shorter than nack timeout
deadline := nackTimeout - 5*time.Second
ctx, cancel := context.WithTimeout(ctx, deadline)
defer cancel()

if err := broker.Ack(evalID, token); err != nil {
	if strings.Contains(err.Error(), "Nack timer expiration") {
		// timer already fired; eval requeued. Make work idempotent.
		log.Printf("late ack for eval %s", evalID)
		return nil
	}
	return err
}

When it happens

Trigger: A worker finishes processing and calls Ack strictly after the nack timeout elapsed — the timer goroutine already ran handleNack, removed the unack entry from the active path, or is mid-expiry when Ack races it.

Common situations: Handler exceeds the nack timeout under load (slow API calls, blocked RPCs); nack timeout configured too aggressively short; long GC pauses or scheduling starvation delaying the ack past the deadline; serialized duplicate job evals waiting behind the delivery limit.

Related errors

• Evaluation ID not found
• Token does not match for Evaluation ID
• eval broker disabled
• timeout cannot be negative
• Failed to start workers: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ee68abb42b2bef6f. Report an issue: GitHub.