{"record":{"id":"6a634cd8abd8a558","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later-6a634c","errorCode":null,"errorMessage":"all servers should be running version %v or later to use JWT ACL auth methods","messagePattern":"all servers should be running version (.+?) or later to use JWT ACL auth methods","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/acl_endpoint.go","lineNumber":2941,"sourceCode":"\n\t// Perform the initial forwarding within the region. This ensures we\n\t// respect stale queries.\n\tif done, err := a.srv.forward(structs.ACLLoginRPCMethod, args, args, reply); done {\n\t\treturn err\n\t}\n\n\t// Measure the login endpoint performance.\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"acl\", \"login\"}, time.Now())\n\n\t// This endpoint can only be used once all servers in all federated regions\n\t// have been upgraded to minACLJWTAuthMethodVersion or greater, since JWT Auth\n\t// method was introduced then.\n\tif !a.srv.peersCache.ServersMeetMinimumVersion(\n\t\tpeers.AllRegions,\n\t\tminACLJWTAuthMethodVersion,\n\t\tfalse,\n\t) {\n\t\treturn fmt.Errorf(\n\t\t\t\"all servers should be running version %v or later to use JWT ACL auth methods\",\n\t\t\tminACLJWTAuthMethodVersion,\n\t\t)\n\t}\n\n\t// Validate the request arguments to ensure it contains all the data it\n\t// needs.\n\tif err := args.Validate(); err != nil {\n\t\treturn structs.NewErrRPCCodedf(http.StatusBadRequest, \"invalid login request: %v\", err)\n\t}\n\n\t// Grab a snapshot of the state, so we can query it safely.\n\tstateSnapshot, err := a.srv.fsm.State().Snapshot()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Lookup the auth method from state, so we have the entire object","sourceCodeStart":2923,"sourceCodeEnd":2959,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/acl_endpoint.go#L2923-L2959","documentation":"Thrown by the Login RPC when the cluster's servers do not all meet minACLJWTAuthMethodVersion — JWT-based ACL auth methods require every server in every region to run at least that Nomad version. It is an RPC preflight guard so the mixed-version cluster never partially processes JWT logins.","triggerScenarios":"A Login request with a JWT-type auth method arrives while peersCache.ServersMeetMinimumVersion(peers.AllRegions, minACLJWTAuthMethodVersion, false) reports at least one server below the required version.","commonSituations":"Rolling upgrade in progress, a stale/old server left in the region, or using JWT auth methods against an older cluster (e.g. pre-1.4.x servers) after a client was upgraded.","solutions":["Upgrade all Nomad servers to at least minACLJWTAuthMethodVersion and wait for the rolling upgrade to finish.","Check nomad server members / raft peers for any node still running an older version; remove or upgrade it.","Retry Login after the cluster converges (autoupdate / scheduler version reporting can lag briefly).","Use a non-JWT auth method (e.g. an auth method type supported by the older cluster) as a temporary workaround."],"exampleFix":"// before: login while a 1.3.x server is still in the cluster\nnomad acl login -method=jwt\n// after: upgrade stragglers, then\nnomad version  # ensure every server >= minACLJWTAuthMethodVersion\nnomad acl login -method=jwt","handlingStrategy":"retry","validationCode":"// check server versions before attempting JWT login\nout, _ := exec.Command(\"nomad\", \"server\", \"members\").Output()\nfor _, m := range parseMembers(out) {\n  if versionLessThan(m.Version, \"1.4.0\") { return fmt.Errorf(\"server %s too old\", m.Name) }\n}","typeGuard":null,"tryCatchPattern":"err := client.ACL().Login(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"version\") {\n    time.Sleep(retryInterval) // retry after rolling upgrade converges\n}","preventionTips":["Complete rolling upgrades before enabling JWT auth methods.","Monitor nomad server members for stragglers on old versions.","Pin client tooling versions in sync with server minimum versions."],"tags":["versioning","cluster","acl","jwt","compatibility"],"backgroundTag":"server-version-too-old","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}