{"record":{"id":"009aa852f807974b","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later","errorCode":null,"errorMessage":"All servers should be running version %v or later to use one-time authentication tokens","messagePattern":"All servers should be running version (.+?) or later to use one-time authentication tokens","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/acl_endpoint.go","lineNumber":1077,"sourceCode":"}\n\nfunc (a *ACL) UpsertOneTimeToken(args *structs.OneTimeTokenUpsertRequest, reply *structs.OneTimeTokenUpsertResponse) error {\n\tif !a.srv.config.ACLEnabled {\n\t\treturn aclDisabled\n\t}\n\tif done, err := a.srv.forward(\n\t\t\"ACL.UpsertOneTimeToken\", args, args, reply); done {\n\t\treturn err\n\t}\n\tdefer metrics.MeasureSince(\n\t\t[]string{\"nomad\", \"acl\", \"upsert_one_time_token\"}, time.Now())\n\n\tif !a.srv.peersCache.ServersMeetMinimumVersion(\n\t\ta.srv.Region(),\n\t\tminOneTimeAuthenticationTokenVersion,\n\t\tfalse,\n\t) {\n\t\treturn fmt.Errorf(\n\t\t\t\"All servers should be running version %v or later to use one-time authentication tokens\",\n\t\t\tminOneTimeAuthenticationTokenVersion,\n\t\t)\n\t}\n\n\t// Snapshot the state\n\tstate, err := a.srv.State().Snapshot()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Look up the token; there's no capability check as you can only\n\t// request a OTT for your own ACL token\n\taclToken, err := state.ACLTokenBySecretID(nil, args.AuthToken)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif aclToken == nil {","sourceCodeStart":1059,"sourceCodeEnd":1095,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/acl_endpoint.go#L1059-L1095","documentation":"UpsertOneTimeToken creates a one-time authentication token for a workload, a feature introduced at minOneTimeAuthenticationTokenVersion. Because RPCs are handled cluster-wide, every server in the local region must understand the feature; if any peer is older, the endpoint refuses the request with this error to avoid state that older servers cannot process.","triggerScenarios":"Invoking the ACL OneTimeToken Upsert RPC while at least one server in the region (excluding failing/left nodes per the don't-recheck flag) reports a version below minOneTimeAuthenticationTokenVersion via peersCache.ServersMeetMinimumVersion.","commonSituations":"Rolling upgrade in progress with mixed server versions; a node that failed to upgrade stuck on an old binary; newly joined server still on a prior release.","solutions":["Upgrade all servers in the region to at least minOneTimeAuthenticationTokenVersion, then retry","Check server versions with `nomad server members` and identify the lagging server","Complete or restart a stalled rolling upgrade (fix the server that didn't restart on the new binary)","Temporarily avoid one-time tokens (use standard ACL tokens) until the cluster is homogeneous"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"members, _ := client.Agent().Members()\nfor _, m := range members.Members {\n    if !serverMeetsMinVersion(m.Tags[\"build\"], minVersion) {\n        return fmt.Errorf(\"server %s at %s too old for one-time tokens\", m.Name, m.Tags[\"build\"])\n    }\n}","typeGuard":null,"tryCatchPattern":"resp, err := acl.UpsertOneTimeToken(req)\nif err != nil && strings.Contains(err.Error(), \"All servers should be running version\") {\n    // wait for rolling upgrade to finish, retry with backoff\n    time.AfterFunc(upgradeWait, func() { retryUpsert(req) })\n    return\n}","preventionTips":["Finish rolling upgrades before enabling one-time token features","Monitor server build versions and alert on skew","Pin cluster upgrades so no server lags the minimum feature version","Gate feature usage (one-time tokens) on a version check at deploy time"],"tags":["nomad","acl","version-skew","cluster-upgrade"],"backgroundTag":"server-version-not-supported","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}