{"record":{"id":"1affad6c2f00e693","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later-1affad","errorCode":null,"errorMessage":"all servers should be running version %v or later to use ACL binding rules","messagePattern":"all servers should be running version (.+?) or later to use ACL binding rules","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/acl_endpoint.go","lineNumber":2281,"sourceCode":"\n\tauthErr := a.srv.Authenticate(a.ctx, args)\n\tif done, err := a.srv.forward(structs.ACLUpsertBindingRulesRPCMethod, args, args, reply); done {\n\t\treturn err\n\t}\n\ta.srv.MeasureRPCRate(\"acl\", structs.RateMetricWrite, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"acl\", \"upsert_binding_rules\"}, time.Now())\n\n\t// ACL binding rules can only be used once all servers in all federated\n\t// regions have been upgraded to 1.5.0 or greater.\n\tif !a.srv.peersCache.ServersMeetMinimumVersion(\n\t\tpeers.AllRegions,\n\t\tminACLBindingRuleVersion,\n\t\tfalse,\n\t) {\n\t\treturn fmt.Errorf(\n\t\t\t\"all servers should be running version %v or later to use ACL binding rules\",\n\t\t\tminACLBindingRuleVersion,\n\t\t)\n\t}\n\n\t// Check management level permissions\n\tif aclObj, err := a.srv.ResolveACL(args); err != nil {\n\t\treturn err\n\t} else if !aclObj.IsManagement() {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\n\t// Validate non-zero set of binding rules. This must be done outside the\n\t// validate function as that uses a loop, which will be skipped if the\n\t// length is zero.\n\tif len(args.ACLBindingRules) == 0 {\n\t\treturn structs.NewErrRPCCoded(http.StatusBadRequest, \"must specify as least one binding rule\")\n\t}","sourceCodeStart":2263,"sourceCodeEnd":2299,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/acl_endpoint.go#L2263-L2299","documentation":"ACL binding rules were introduced in Nomad 1.5.0. UpsertBindingRules verifies that every server in every federated region runs at least minACLBindingRuleVersion (1.5.0) before writing a binding rule; if any peer is older, the RPC fails with this message so legacy servers never see the new state.","triggerScenarios":"Creating/updating a binding rule ('nomad acl binding-rule create/update' / ACL().UpsertBindingRules) while at least one server in any federated region is below 1.5.0.","commonSituations":"Rolling upgrade from 1.4.x to 1.5.x still in progress; an operator provisions binding rules for SSO before completing the fleet upgrade; a forgotten lagging region.","solutions":["Upgrade all servers in all federated regions to >= 1.5.0.","Confirm with 'nomad server members' that no server is below 1.5.0.","Retry the UpsertBindingRules request after the upgrade completes.","De-federate or remove regions that will not be upgraded."],"exampleFix":"// before (a server still on 1.4.x)\n_, _, err := nomadClient.ACL().UpsertBindingRules(ctx, &api.ACLBindingRulesUpsertRequest{BindingRules: []*api.ACLBindingRule{br}}, nil)\n// -> all servers should be running version 1.5.0 or later to use ACL binding rules\n// after (all servers >= 1.5.0)\n_, _, err := nomadClient.ACL().UpsertBindingRules(ctx, &api.ACLBindingRulesUpsertRequest{BindingRules: []*api.ACLBindingRule{br}}, nil)","handlingStrategy":"retry","validationCode":"members, _, _ := client.Agent().Members()\nfor _, m := range members.Members {\n    if compareVersion(m.Tags[\"version\"], \"1.5.0\") < 0 {\n        return fmt.Errorf(\"server %s at %s; binding rules require >= 1.5.0\", m.Name, m.Tags[\"version\"])\n    }\n}","typeGuard":"func supportsBindingRules(v string) bool { c, err := version.NewVersion(v); min, _ := version.NewVersion(\"1.5.0\"); return err == nil && c.GreaterThanOrEqual(min) }","tryCatchPattern":"_, _, err := client.ACL().UpsertBindingRules(ctx, req, nil)\nif err != nil && strings.Contains(err.Error(), \"should be running version\") {\n    time.Sleep(30 * time.Second)\n    return retryUpsertBindingRules(req)\n}","preventionTips":["Provision binding rules only after all servers reach 1.5.0","Verify federation-wide versions before SSO setup automation","Keep lagging regions out of federation or upgrade them first"],"tags":["nomad","acl","version-skew","federation","binding-rules"],"backgroundTag":"cluster-version-skew","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}