hashicorp/nomad · error

eval broker is enabled; eval broker must be paused to delete

Error message

eval broker is enabled; eval broker must be paused to delete evals

What it means

DeleteEvals is a destructive, user-initiated operation that must run while the eval broker is paused so in-flight scheduling state is frozen. The state store validates scheduler config inside the write txn and rejects the delete if PauseEvalBroker is not set.

Source

Thrown at nomad/state/state_store.go:3604

	if err := txn.Insert("index", &IndexEntry{"evals", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}
	return nil
}

// DeleteEvalsByFilter is used to delete all evals that are both safe to delete
// and match a filter.
func (s *StateStore) DeleteEvalsByFilter(index uint64, filterExpr string, pageToken string, perPage int32) error {
	txn := s.db.WriteTxn(index)
	defer txn.Abort()

	// These are always user-initiated, so ensure the eval broker is paused.
	_, schedConfig, err := s.schedulerConfigTxn(txn)
	if err != nil {
		return err
	}
	if schedConfig == nil || !schedConfig.PauseEvalBroker {
		return errors.New("eval broker is enabled; eval broker must be paused to delete evals")
	}

	filter, err := bexpr.CreateEvaluator(filterExpr)
	if err != nil {
		return err
	}

	iter, err := s.Evals(nil, SortDefault)
	if err != nil {
		return fmt.Errorf("failed to lookup evals: %v", err)
	}

	// Note: Paginator imports this package for testing so we can't just use
	// Paginator
	pageCount := int32(0)

	for {
		if pageCount >= perPage {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Pause the eval broker first: `nomad operator scheduler set-config -pause-eval-broker=true`, then retry the deletion
2. Delete evals again after cleanup and unpause with -pause-eval-broker=false
3. If using the HTTP API, PUT /v1/operator/scheduler/configuration with PauseEvalBroker:true before deleting

Example fix

// before
nomad system gc  # deletes evals with broker running -> error
// after
nomad operator scheduler set-config -pause-eval-broker=true
nomad system gc
nomad operator scheduler set-config -pause-eval-broker=false

cfg, _, _ := client.Operator().SchedulerGetConfiguration(nil)
if cfg.SchedulerConfig == nil || !cfg.SchedulerConfig.PauseEvalBroker {
    _, _, err := client.Operator().SchedulerSetConfiguration(
        (&api.SchedulerConfiguration{PauseEvalBroker: true}).Canonicalize(), nil, nil)
    if err != nil { return err }
    defer unpauseEvalBroker(client)
}
// proceed with eval deletion

func brokerPaused(c *api.SchedulerConfiguration) bool {
    return c != nil && c.PauseEvalBroker
}

if err := deleteEvals(client, filter); err != nil && strings.Contains(err.Error(), "eval broker must be paused") {
    pauseBroker(client); defer unpauseBroker(client)
    err = deleteEvals(client, filter)
}

When it happens

Trigger: Calling StateStore.DeleteEvals (exposed via `nomad system gc`-style eval deletion or the evals DELETE API) while the scheduler config has PauseEvalBroker=false or is nil.

Common situations: Operators running bulk eval cleanup during incident triage without first pausing the broker; automation that deletes evals as part of garbage collection; mistaken assumption deletion can happen live.

Related errors

• setting job status failed: %v
• job summary lookup failed: %v
• eval delete failed: %v
• failed to lookup job for eval: %v
• failed to lookup eval allocs: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e2bed141a09d99e0. Report an issue: GitHub.