{"record":{"id":"25866c259f2a77a4","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later-25866c","errorCode":null,"errorMessage":"all servers should be running version %v or later to use client intro tokens","messagePattern":"all servers should be running version (.+?) or later to use client intro tokens","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/acl_endpoint.go","lineNumber":3202,"sourceCode":"\t}\n\treturn j, nil\n}\n\nfunc (a *ACL) CreateClientIntroductionToken(\n\targs *structs.ACLCreateClientIntroductionTokenRequest,\n\treply *structs.ACLCreateClientIntroductionTokenResponse) error {\n\n\tauthErr := a.srv.Authenticate(a.ctx, args)\n\n\tif done, err := a.srv.forward(structs.ACLCreateClientIntroductionTokenRPCMethod, args, args, reply); done {\n\t\treturn err\n\t}\n\ta.srv.MeasureRPCRate(\"acl\", structs.RateMetricWrite, args)\n\n\t// This endpoint can only be used once all servers in the local region have\n\t// been upgraded to minVersionNodeIntro or greater.\n\tif !a.srv.peersCache.ServersMeetMinimumVersion(a.srv.Region(), minVersionNodeIntro, false) {\n\t\treturn fmt.Errorf(\n\t\t\t\"all servers should be running version %v or later to use client intro tokens\",\n\t\t\tminVersionNodeIntro)\n\t}\n\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\n\t\t\"nomad\", \"acl\", \"create_node_introduction_identity\"}, time.Now())\n\n\t// Unlike the other ACL RPCs, this accepts node write permissions rather\n\t// than management. This allows cluster administrators to delegate node\n\t// introduction identity operations to other users who can bring their own\n\t// nodes to join the cluster.\n\tif aclObj, err := a.srv.ResolveACL(args); err != nil {\n\t\treturn err\n\t} else if !aclObj.AllowNodeWrite() {\n\t\treturn structs.ErrPermissionDenied","sourceCodeStart":3184,"sourceCodeEnd":3220,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/acl_endpoint.go#L3184-L3220","documentation":"Nomad's CreateClientIntroductionToken RPC can only issue client introduction tokens once every server in the local region is running at least the minVersionNodeIntro release. The server checks its peersCache via ServersMeetMinimumVersion and rejects the request with this message if any peer is older. It is a cluster-federation/upgrade-consistency guard, not a client bug.","triggerScenarios":"Calling the ClientIntroductionToken RPC (or agent APIs that use it, e.g. node introduction flows) while at least one server in the local region runs a Nomad version older than minVersionNodeIntro. The check happens after rate measurement and before auth error evaluation.","commonSituations":"Rolling upgrade still in progress; a lagging server failed to upgrade and was left in the peer list; mixed-version multi-region cluster where one region was not upgraded; server rejoined the gossip pool with old binary after a rollback.","solutions":["Upgrade all servers in the region to at least the required version (minVersionNodeIntro) and wait for the upgrade to finish","Verify with `nomad server members` / `nomad version` that every server reports the minimum version","Remove or restart any stale/outdated server from the region's peer list, then retry","If the upgrade is intentional and ongoing, retry the request after the rolling upgrade completes"],"exampleFix":"// before: mixing versions during rolling upgrade\n// after: confirm all servers upgraded\n$ nomad server members  # all servers >= minVersionNodeIntro\n$ nomad acl token create ... # retry client intro token request","handlingStrategy":"validation","validationCode":"members, _ := agentClient.Agent().Members()\nrequiredVersion := \"1.9.0\" // minVersionNodeIntro\nfor _, m := range members.Members {\n    if v, err := version.NewVersion(m.Tags[\"build\"]); err != nil || v.LessThan(required) {\n        return fmt.Errorf(\"server %s below %s, intro tokens unavailable\", m.Name, requiredVersion)\n    }\n}","typeGuard":null,"tryCatchPattern":"var versionSkewErr = regexp.MustCompile(`all servers should be running version`)\nif versionSkewErr.MatchString(err.Error()) {\n    // defer token issuance until the rolling upgrade completes\n}","preventionTips":["Complete rolling server upgrades before enabling intro-token features","Monitor `nomad server members` versions in CI/CD before feature rollout","Remove decommissioned/outdated servers from the region promptly"],"tags":["nomad","cluster-upgrade","version-skew","rpc"],"backgroundTag":"mixed-version-cluster","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}