hashicorp/nomad · error

all servers should be running version %v or later to use ACL

Error message

all servers should be running version %v or later to use ACL roles

What it means

Nomad ACL Roles were introduced in Nomad 1.4.0. Before any ACL role RPC is accepted, the server checks that every server in every federated region is running at least minACLRoleVersion (1.4.0). If any peer is older, the entire cluster is considered not yet upgraded for roles and the write is rejected with this message, preventing mixed-version state corruption of the new ACL schema.

Source

Thrown at nomad/acl_endpoint.go:1258

	}
	authErr := a.srv.Authenticate(a.ctx, args)
	// This endpoint always forwards to the authoritative region as ACL roles
	// are global.
	args.Region = a.srv.config.AuthoritativeRegion

	if done, err := a.srv.forward(structs.ACLUpsertRolesRPCMethod, args, args, reply); done {
		return err
	}
	a.srv.MeasureRPCRate("acl", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "acl", "upsert_roles"}, time.Now())

	// ACL roles can only be used once all servers, in all federated regions
	// have been upgraded to 1.4.0 or greater.
	if !a.srv.peersCache.ServersMeetMinimumVersion(peers.AllRegions, minACLRoleVersion, false) {
		return fmt.Errorf("all servers should be running version %v or later to use ACL roles",
			minACLRoleVersion)
	}

	// Only management level permissions can create ACL roles.
	if aclObj, err := a.srv.ResolveACL(args); err != nil {
		return err
	} else if !aclObj.IsManagement() {
		return structs.ErrPermissionDenied
	}

	// Snapshot the state so we can perform lookups against the ID and policy
	// links if needed. Do it here, so we only need to do this once no matter
	// how many roles we are upserting.
	stateSnapshot, err := a.srv.State().Snapshot()
	if err != nil {
		return err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all Nomad servers in every federated region to 1.4.0 or later (rolling upgrade: first group, then a second group, leader last).
2. Verify version parity with 'nomad server members' or the /v1/agent/members endpoint across all regions.
3. If a stale region can never be upgraded, remove it from federation before creating ACL roles.
4. Retry the UpsertRoles RPC once the whole fleet reports >= 1.4.0.

Example fix

// before (mixed fleet, 1.3.x server present)
nomadClient.ACL().UpsertRoles(ctx, &api.ACLRolesUpsertRequest{Roles: []*api.ACLRole{role}}, nil)
// -> all servers should be running version 1.4.0 or later to use ACL roles
// after (all servers >= 1.4.0)
// nomad server members  # confirm every server is 1.4.0+
nomadClient.ACL().UpsertRoles(ctx, &api.ACLRolesUpsertRequest{Roles: []*api.ACLRole{role}}, nil)

// before calling UpsertRoles
members, _, _ := client.Agent().Members()
for _, m := range members.Members {
    if m.Tags["rpc"] != "" && compareVersion(m.Tags["version"], "1.4.0") < 0 {
        return fmt.Errorf("server %s is %s; ACL roles require >= 1.4.0", m.Name, m.Tags["version"])
    }
}

func supportsACLRoles(v string) bool { c, err := version.NewVersion(v); min, _ := version.NewVersion("1.4.0"); return err == nil && c.GreaterThanOrEqual(min) }

err := client.ACL().UpsertRoles(ctx, req, nil)
if err != nil && strings.Contains(err.Error(), "should be running version") {
    // cluster mid-upgrade: wait and retry
    time.Sleep(30 * time.Second)
    return retryUpsertRoles(req)
}

When it happens

Trigger: Calling ACL role create/update via UpsertRoles (client. ACL().UpsertRoles or 'nomad acl role create/update') while at least one server in any federated region is running a version below 1.4.0; typically during a rolling upgrade.

Common situations: Operator upgrades only some servers in a multi-region/federated cluster and then attempts role management; a lagging region still runs 1.3.x; a long- forgotten test region participates in federation.

Related errors

• all servers should be running version %v or later to use ACL
• all servers should be running version %v or later to use ACL
• All servers should be running version %v or later to use one
• tasks can only have 1 identity block until all servers are u
• ACL policy not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e628e7d2d788d377. Report an issue: GitHub.