hashicorp/nomad · error

all servers should be running version %v or later to use ACL

Error message

all servers should be running version %v or later to use ACL auth methods

What it means

ACL auth methods (SSO/OIDC) require Nomad 1.5.0 or later. UpsertAuthMethods checks that every server in every federated region satisfies minACLAuthMethodVersion (1.5.0) before persisting an auth method, and rejects the write with this message otherwise, protecting the cluster from mixed-version ACL schema handling.

Source

Thrown at nomad/acl_endpoint.go:1902

	if !a.srv.config.ACLEnabled {
		return aclDisabled
	}
	authErr := a.srv.Authenticate(a.ctx, args)
	args.Region = a.srv.config.AuthoritativeRegion

	if done, err := a.srv.forward(structs.ACLUpsertAuthMethodsRPCMethod, args, args, reply); done {
		return err
	}
	a.srv.MeasureRPCRate("acl", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "acl", "upsert_auth_methods"}, time.Now())

	// ACL auth methods can only be used once all servers in all federated
	// regions have been upgraded to 1.5.0 or greater.
	if !a.srv.peersCache.ServersMeetMinimumVersion(peers.AllRegions, minACLAuthMethodVersion, false) {
		return fmt.Errorf("all servers should be running version %v or later to use ACL auth methods",
			minACLAuthMethodVersion)
	}

	// Check management level permissions
	if aclObj, err := a.srv.ResolveACL(args); err != nil {
		return err
	} else if !aclObj.IsManagement() {
		return structs.ErrPermissionDenied
	}

	// Validate non-zero set of auth methods
	if len(args.AuthMethods) == 0 {
		return structs.NewErrRPCCoded(http.StatusBadRequest, "must specify as least one auth method")
	}

	// Snapshot the state so we can make lookups to verify default method
	stateSnapshot, err := a.srv.State().Snapshot()
	if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all servers in all federated regions to >= 1.5.0.
2. Confirm with 'nomad server members' that no server is below 1.5.0.
3. Re-run the auth-method create/update after the fleet is upgraded.
4. Exclude or de-federate regions that will not be upgraded.

Example fix

// before (a server at 1.4.x remains)
_, _, err := nomadClient.ACL().UpsertAuthMethods(ctx, &api.ACLAuthMethodsUpsertRequest{AuthMethods: []*api.ACLAuthMethod{am}}, nil)
// -> all servers should be running version 1.5.0 or later to use ACL auth methods
// after (all servers >= 1.5.0)
_, _, err := nomadClient.ACL().UpsertAuthMethods(ctx, &api.ACLAuthMethodsUpsertRequest{AuthMethods: []*api.ACLAuthMethod{am}}, nil)

members, _, _ := client.Agent().Members()
for _, m := range members.Members {
    if compareVersion(m.Tags["version"], "1.5.0") < 0 {
        return fmt.Errorf("server %s at %s; auth methods require >= 1.5.0", m.Name, m.Tags["version"])
    }
}

func supportsAuthMethods(v string) bool { c, err := version.NewVersion(v); min, _ := version.NewVersion("1.5.0"); return err == nil && c.GreaterThanOrEqual(min) }

_, _, err := client.ACL().UpsertAuthMethods(ctx, req, nil)
if err != nil && strings.Contains(err.Error(), "should be running version") {
    time.Sleep(30 * time.Second)
    return retryUpsertAuthMethods(req)
}

When it happens

Trigger: Creating or updating an ACL auth method ('nomad acl auth-method create/update' / ACL().UpsertAuthMethods) while any server in any federated region is below 1.5.0.

Common situations: Rolling upgrade from 1.4.x to 1.5.x not yet complete; CI automation that provisions SSO runs before the fleet is upgraded; one region left behind at 1.4.x.

Related errors

• all servers should be running version %v or later to use ACL
• all servers should be running version %v or later to use ACL
• missing OIDCDiscoveryURL
• missing OIDCClientID
• missing auth method name

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9002477cc9b7b0ae. Report an issue: GitHub.