hashicorp/nomad · error

all servers must be running version %v or later to delete ev

Error message

all servers must be running version %v or later to delete evals by filter

What it means

Returned by Evaluations.Delete when a filter-based delete is requested but not every server in the region meets the minimum version that supports eval delete-by-filter. The operation requires cluster-wide support because the filter is evaluated consistently across servers.

Source

Thrown at nomad/eval_endpoint.go:470

		return err
	}
	e.srv.MeasureRPCRate("eval", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "eval", "delete"}, time.Now())

	// This RPC endpoint is very destructive and alters Nomad's core state,
	// meaning only those with management tokens can call it.
	if aclObj, err := e.srv.ResolveACL(args); err != nil {
		return err
	} else if !aclObj.IsManagement() {
		return structs.ErrPermissionDenied
	}

	if args.Filter != "" && !e.srv.peersCache.ServersMeetMinimumVersion(
		e.srv.Region(), minVersionEvalDeleteByFilter, true) {
		return fmt.Errorf(
			"all servers must be running version %v or later to delete evals by filter",
			minVersionEvalDeleteByFilter)
	}
	if args.Filter != "" && len(args.EvalIDs) > 0 {
		return fmt.Errorf("evals cannot be deleted by both ID and filter")
	}
	if args.Filter == "" && len(args.EvalIDs) == 0 {
		return fmt.Errorf("evals must be deleted by either ID or filter")
	}

	// The eval broker must be disabled otherwise Nomad's state will likely get
	// wild in a very un-fun way.
	if e.srv.evalBroker.Enabled() {
		return errors.New("eval broker is enabled; eval broker must be paused to delete evals")
	}

	if args.Filter != "" {
		count, index, err := e.deleteEvalsByFilter(args)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all servers in the region to at least minVersionEvalDeleteByFilter
2. Delete evals by explicit IDs instead of a filter until the cluster is fully upgraded
3. Verify server versions (nomad server members / /v1/agent/members) before retrying

Example fix

// before
req := &api.EvalDeleteRequest{Filter: "Status == complete"}
// after
req := &api.EvalDeleteRequest{EvalIDs: []string{evalID}} // until all servers upgraded

members, _, err := client.Agent().Members()
if err != nil { return err }
for _, m := range members.Members {
    if version.Compare(m.Tags["build"], "0.11.5") < 0 { return fmt.Errorf("server %s too old", m.Name) }
}

When it happens

Trigger: Calling the eval delete API with a non-empty args.Filter while e.srv.peersCache.ServersMeetMinimumVersion reports some server in the region is older than minVersionEvalDeleteByFilter.

Common situations: Rolling upgrades where one or more servers still run an older Nomad version; forgetting to upgrade all client servers (not just leaders) before using filter-based eval cleanup APIs.

Related errors

• all servers should be running version %v or later to use cli
• all servers must be running version %v or later to apply var
• no servers
• missing NodeID
• missing AllocID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f71916196957fcb2. Report an issue: GitHub.