all servers should be running version %v or later to use ACL

Error message

all servers should be running version %v or later to use ACL binding rules

What it means

ACL binding rules were introduced in Nomad 1.5.0. UpsertBindingRules verifies that every server in every federated region runs at least minACLBindingRuleVersion (1.5.0) before writing a binding rule; if any peer is older, the RPC fails with this message so legacy servers never see the new state.

Source

Thrown at nomad/acl_endpoint.go:2281

	authErr := a.srv.Authenticate(a.ctx, args)
	if done, err := a.srv.forward(structs.ACLUpsertBindingRulesRPCMethod, args, args, reply); done {
		return err
	}
	a.srv.MeasureRPCRate("acl", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "acl", "upsert_binding_rules"}, time.Now())

	// ACL binding rules can only be used once all servers in all federated
	// regions have been upgraded to 1.5.0 or greater.
	if !a.srv.peersCache.ServersMeetMinimumVersion(
		peers.AllRegions,
		minACLBindingRuleVersion,
		false,
	) {
		return fmt.Errorf(
			"all servers should be running version %v or later to use ACL binding rules",
			minACLBindingRuleVersion,
		)
	}

	// Check management level permissions
	if aclObj, err := a.srv.ResolveACL(args); err != nil {
		return err
	} else if !aclObj.IsManagement() {
		return structs.ErrPermissionDenied
	}

	// Validate non-zero set of binding rules. This must be done outside the
	// validate function as that uses a loop, which will be skipped if the
	// length is zero.
	if len(args.ACLBindingRules) == 0 {
		return structs.NewErrRPCCoded(http.StatusBadRequest, "must specify as least one binding rule")
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Upgrade all servers in all federated regions to >= 1.5.0.
Confirm with 'nomad server members' that no server is below 1.5.0.
Retry the UpsertBindingRules request after the upgrade completes.
De-federate or remove regions that will not be upgraded.

Example fix

// before (a server still on 1.4.x)
_, _, err := nomadClient.ACL().UpsertBindingRules(ctx, &api.ACLBindingRulesUpsertRequest{BindingRules: []*api.ACLBindingRule{br}}, nil)
// -> all servers should be running version 1.5.0 or later to use ACL binding rules
// after (all servers >= 1.5.0)
_, _, err := nomadClient.ACL().UpsertBindingRules(ctx, &api.ACLBindingRulesUpsertRequest{BindingRules: []*api.ACLBindingRule{br}}, nil)

Defensive patterns

Strategy: retry

Validate before calling

members, _, _ := client.Agent().Members()
for _, m := range members.Members {
    if compareVersion(m.Tags["version"], "1.5.0") < 0 {
        return fmt.Errorf("server %s at %s; binding rules require >= 1.5.0", m.Name, m.Tags["version"])
    }
}

Type guard

func supportsBindingRules(v string) bool { c, err := version.NewVersion(v); min, _ := version.NewVersion("1.5.0"); return err == nil && c.GreaterThanOrEqual(min) }

Try / catch

_, _, err := client.ACL().UpsertBindingRules(ctx, req, nil)
if err != nil && strings.Contains(err.Error(), "should be running version") {
    time.Sleep(30 * time.Second)
    return retryUpsertBindingRules(req)
}

Prevention

Provision binding rules only after all servers reach 1.5.0
Verify federation-wide versions before SSO setup automation
Keep lagging regions out of federation or upgrade them first

When it happens

Trigger: Creating/updating a binding rule ('nomad acl binding-rule create/update' / ACL().UpsertBindingRules) while at least one server in any federated region is below 1.5.0.

Common situations: Rolling upgrade from 1.4.x to 1.5.x still in progress; an operator provisions binding rules for SSO before completing the fleet upgrade; a forgotten lagging region.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1affad6c2f00e693. Report an issue: GitHub.