{"record":{"id":"9002477cc9b7b0ae","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later-900247","errorCode":null,"errorMessage":"all servers should be running version %v or later to use ACL auth methods","messagePattern":"all servers should be running version (.+?) or later to use ACL auth methods","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/acl_endpoint.go","lineNumber":1902,"sourceCode":"\tif !a.srv.config.ACLEnabled {\n\t\treturn aclDisabled\n\t}\n\tauthErr := a.srv.Authenticate(a.ctx, args)\n\targs.Region = a.srv.config.AuthoritativeRegion\n\n\tif done, err := a.srv.forward(structs.ACLUpsertAuthMethodsRPCMethod, args, args, reply); done {\n\t\treturn err\n\t}\n\ta.srv.MeasureRPCRate(\"acl\", structs.RateMetricWrite, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"acl\", \"upsert_auth_methods\"}, time.Now())\n\n\t// ACL auth methods can only be used once all servers in all federated\n\t// regions have been upgraded to 1.5.0 or greater.\n\tif !a.srv.peersCache.ServersMeetMinimumVersion(peers.AllRegions, minACLAuthMethodVersion, false) {\n\t\treturn fmt.Errorf(\"all servers should be running version %v or later to use ACL auth methods\",\n\t\t\tminACLAuthMethodVersion)\n\t}\n\n\t// Check management level permissions\n\tif aclObj, err := a.srv.ResolveACL(args); err != nil {\n\t\treturn err\n\t} else if !aclObj.IsManagement() {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\n\t// Validate non-zero set of auth methods\n\tif len(args.AuthMethods) == 0 {\n\t\treturn structs.NewErrRPCCoded(http.StatusBadRequest, \"must specify as least one auth method\")\n\t}\n\n\t// Snapshot the state so we can make lookups to verify default method\n\tstateSnapshot, err := a.srv.State().Snapshot()\n\tif err != nil {","sourceCodeStart":1884,"sourceCodeEnd":1920,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/acl_endpoint.go#L1884-L1920","documentation":"ACL auth methods (SSO/OIDC) require Nomad 1.5.0 or later. UpsertAuthMethods checks that every server in every federated region satisfies minACLAuthMethodVersion (1.5.0) before persisting an auth method, and rejects the write with this message otherwise, protecting the cluster from mixed-version ACL schema handling.","triggerScenarios":"Creating or updating an ACL auth method ('nomad acl auth-method create/update' / ACL().UpsertAuthMethods) while any server in any federated region is below 1.5.0.","commonSituations":"Rolling upgrade from 1.4.x to 1.5.x not yet complete; CI automation that provisions SSO runs before the fleet is upgraded; one region left behind at 1.4.x.","solutions":["Upgrade all servers in all federated regions to >= 1.5.0.","Confirm with 'nomad server members' that no server is below 1.5.0.","Re-run the auth-method create/update after the fleet is upgraded.","Exclude or de-federate regions that will not be upgraded."],"exampleFix":"// before (a server at 1.4.x remains)\n_, _, err := nomadClient.ACL().UpsertAuthMethods(ctx, &api.ACLAuthMethodsUpsertRequest{AuthMethods: []*api.ACLAuthMethod{am}}, nil)\n// -> all servers should be running version 1.5.0 or later to use ACL auth methods\n// after (all servers >= 1.5.0)\n_, _, err := nomadClient.ACL().UpsertAuthMethods(ctx, &api.ACLAuthMethodsUpsertRequest{AuthMethods: []*api.ACLAuthMethod{am}}, nil)","handlingStrategy":"retry","validationCode":"members, _, _ := client.Agent().Members()\nfor _, m := range members.Members {\n    if compareVersion(m.Tags[\"version\"], \"1.5.0\") < 0 {\n        return fmt.Errorf(\"server %s at %s; auth methods require >= 1.5.0\", m.Name, m.Tags[\"version\"])\n    }\n}","typeGuard":"func supportsAuthMethods(v string) bool { c, err := version.NewVersion(v); min, _ := version.NewVersion(\"1.5.0\"); return err == nil && c.GreaterThanOrEqual(min) }","tryCatchPattern":"_, _, err := client.ACL().UpsertAuthMethods(ctx, req, nil)\nif err != nil && strings.Contains(err.Error(), \"should be running version\") {\n    time.Sleep(30 * time.Second)\n    return retryUpsertAuthMethods(req)\n}","preventionTips":["Upgrade the whole fleet to 1.5.0 before provisioning SSO auth methods","Check federation members in every region, not just the local one","Automate version checks before IaC applies auth-method changes"],"tags":["nomad","acl","oidc","version-skew","federation"],"backgroundTag":"cluster-version-skew","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}