{"record":{"id":"e628e7d2d788d377","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later-e628e7","errorCode":null,"errorMessage":"all servers should be running version %v or later to use ACL roles","messagePattern":"all servers should be running version (.+?) or later to use ACL roles","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/acl_endpoint.go","lineNumber":1258,"sourceCode":"\t}\n\tauthErr := a.srv.Authenticate(a.ctx, args)\n\t// This endpoint always forwards to the authoritative region as ACL roles\n\t// are global.\n\targs.Region = a.srv.config.AuthoritativeRegion\n\n\tif done, err := a.srv.forward(structs.ACLUpsertRolesRPCMethod, args, args, reply); done {\n\t\treturn err\n\t}\n\ta.srv.MeasureRPCRate(\"acl\", structs.RateMetricWrite, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"acl\", \"upsert_roles\"}, time.Now())\n\n\t// ACL roles can only be used once all servers, in all federated regions\n\t// have been upgraded to 1.4.0 or greater.\n\tif !a.srv.peersCache.ServersMeetMinimumVersion(peers.AllRegions, minACLRoleVersion, false) {\n\t\treturn fmt.Errorf(\"all servers should be running version %v or later to use ACL roles\",\n\t\t\tminACLRoleVersion)\n\t}\n\n\t// Only management level permissions can create ACL roles.\n\tif aclObj, err := a.srv.ResolveACL(args); err != nil {\n\t\treturn err\n\t} else if !aclObj.IsManagement() {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\n\t// Snapshot the state so we can perform lookups against the ID and policy\n\t// links if needed. Do it here, so we only need to do this once no matter\n\t// how many roles we are upserting.\n\tstateSnapshot, err := a.srv.State().Snapshot()\n\tif err != nil {\n\t\treturn err\n\t}\n","sourceCodeStart":1240,"sourceCodeEnd":1276,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/acl_endpoint.go#L1240-L1276","documentation":"Nomad ACL Roles were introduced in Nomad 1.4.0. Before any ACL role RPC is accepted, the server checks that every server in every federated region is running at least minACLRoleVersion (1.4.0). If any peer is older, the entire cluster is considered not yet upgraded for roles and the write is rejected with this message, preventing mixed-version state corruption of the new ACL schema.","triggerScenarios":"Calling ACL role create/update via UpsertRoles (client. ACL().UpsertRoles or 'nomad acl role create/update') while at least one server in any federated region is running a version below 1.4.0; typically during a rolling upgrade.","commonSituations":"Operator upgrades only some servers in a multi-region/federated cluster and then attempts role management; a lagging region still runs 1.3.x; a long- forgotten test region participates in federation.","solutions":["Upgrade all Nomad servers in every federated region to 1.4.0 or later (rolling upgrade: first group, then a second group, leader last).","Verify version parity with 'nomad server members' or the /v1/agent/members endpoint across all regions.","If a stale region can never be upgraded, remove it from federation before creating ACL roles.","Retry the UpsertRoles RPC once the whole fleet reports >= 1.4.0."],"exampleFix":"// before (mixed fleet, 1.3.x server present)\nnomadClient.ACL().UpsertRoles(ctx, &api.ACLRolesUpsertRequest{Roles: []*api.ACLRole{role}}, nil)\n// -> all servers should be running version 1.4.0 or later to use ACL roles\n// after (all servers >= 1.4.0)\n// nomad server members  # confirm every server is 1.4.0+\nnomadClient.ACL().UpsertRoles(ctx, &api.ACLRolesUpsertRequest{Roles: []*api.ACLRole{role}}, nil)","handlingStrategy":"retry","validationCode":"// before calling UpsertRoles\nmembers, _, _ := client.Agent().Members()\nfor _, m := range members.Members {\n    if m.Tags[\"rpc\"] != \"\" && compareVersion(m.Tags[\"version\"], \"1.4.0\") < 0 {\n        return fmt.Errorf(\"server %s is %s; ACL roles require >= 1.4.0\", m.Name, m.Tags[\"version\"])\n    }\n}","typeGuard":"func supportsACLRoles(v string) bool { c, err := version.NewVersion(v); min, _ := version.NewVersion(\"1.4.0\"); return err == nil && c.GreaterThanOrEqual(min) }","tryCatchPattern":"err := client.ACL().UpsertRoles(ctx, req, nil)\nif err != nil && strings.Contains(err.Error(), \"should be running version\") {\n    // cluster mid-upgrade: wait and retry\n    time.Sleep(30 * time.Second)\n    return retryUpsertRoles(req)\n}","preventionTips":["Complete Nomad rolling upgrades before mutating new-generation ACL objects","Monitor 'nomad server members' versions across all federated regions","Gate ACL-role automation on cluster version checks","Document that roles/auth-methods/binding-rules need fleet-wide version parity"],"tags":["nomad","acl","version-skew","federation","rolling-upgrade"],"backgroundTag":"cluster-version-skew","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}