hashicorp/nomad · warning

eval broker disabled

Error message

eval broker disabled

What it means

Dequeue cannot serve evaluations because the eval broker is disabled; scanForSchedulers returns this sentinel error when b.enabled is false. A broker is disabled until the leader completes setup (SetEnabled(true)) or after leadership is lost. Callers treat it as 'nothing to dequeue yet', not a fatal condition.

Source

Thrown at nomad/eval_broker.go:441

	}

	// Block until we get work
	scan := b.waitForSchedulers(schedulers, timeoutCh)
	if scan {
		goto SCAN
	}
	return nil, "", nil
}

// scanForSchedulers scans for work on any of the schedulers. The highest priority work
// is dequeued first. This may return nothing if there is no work waiting.
func (b *EvalBroker) scanForSchedulers(schedulers []string) (*structs.Evaluation, string, error) {
	b.l.Lock()
	defer b.l.Unlock()

	// Do nothing if not enabled
	if !b.enabled {
		return nil, "", fmt.Errorf("eval broker disabled")
	}

	// Scan for eligible work
	var eligibleSched []string
	var eligiblePriority int
	for _, sched := range schedulers {
		// Get the ready queue for this scheduler
		readyQueue, ok := b.ready[sched]
		if !ok {
			continue
		}

		// Peek at the next item
		ready := readyQueue.Peek()
		if ready == nil {
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Wait for the server to become leader — Dequeue is only meaningful on the active leader.
2. In tests, call broker.SetEnabled(true) before Dequeue.
3. Handle this error by retrying after a backoff or shutting down the dequeue loop, since it signals the broker is intentionally offline.
4. Verify the server hasn't shut down / lost leadership before dequeuing.

Example fix

// before
if err != nil { return err }
// after
eval, token, err := broker.Dequeue(schedulers, timeout)
if err != nil && err.Error() == "eval broker disabled" {
	time.Sleep(time.Second) // broker offline; retry later
	return nil
}

// caller-side check where possible
if !server.IsLeader() {
	return nil // don't dequeue on non-leaders
}

eval, token, err := broker.Dequeue(scheds, timeout)
if err != nil {
	if strings.Contains(err.Error(), "eval broker disabled") {
		time.Sleep(time.Second) // broker not enabled yet / shut down
		continue
	}
	return err
}

When it happens

Trigger: Calling Dequeue (→ scanForSchedulers) while the broker is disabled: before the server becomes leader, after step-down/shutdown, or in unit tests constructing a broker without calling SetEnabled(true).

Common situations: A worker/agent polling Dequeue during a leader election window; tests that create a broker via NewEvalBroker but never enable it; querying after server shutdown while the broker handle is still held.

Related errors

• Evaluation ID not found
• Token does not match for Evaluation ID
• Evaluation ID Ack'd after Nack timer expiration
• users: release of unused uid/gid
• timeout cannot be negative

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ebb1483c35584d4f. Report an issue: GitHub.