{"record":{"id":"01a52896ddf7de26","repo":"hashicorp/nomad","slug":"all-servers-must-be-running-version-v-or-later-to-01a528","errorCode":null,"errorMessage":"all servers must be running version %v or later to upsert node pools","messagePattern":"all servers must be running version (.+?) or later to upsert node pools","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/node_pool_endpoint.go","lineNumber":195,"sourceCode":"\t// request.\n\taclObj, err := n.srv.ResolveACL(args)\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, pool := range args.NodePools {\n\t\tif !aclObj.AllowNodePoolOperation(pool.Name, acl.NodePoolCapabilityWrite) {\n\t\t\treturn structs.ErrPermissionDenied\n\t\t}\n\n\t\t// Strict enforcement for write requests.\n\t\t// If not licensed then requests will be denied.\n\t\tif err := n.validateLicense(pool); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tif !n.srv.peersCache.ServersMeetMinimumVersion(n.srv.Region(), minNodePoolsVersion, true) {\n\t\treturn fmt.Errorf(\"all servers must be running version %v or later to upsert node pools\", minNodePoolsVersion)\n\t}\n\n\t// Validate request.\n\tif len(args.NodePools) == 0 {\n\t\treturn structs.NewErrRPCCodedf(http.StatusBadRequest, \"must specify at least one node pool\")\n\t}\n\tfor _, pool := range args.NodePools {\n\n\t\tpool.Canonicalize()\n\n\t\tif err := pool.Validate(); err != nil {\n\t\t\treturn structs.NewErrRPCCodedf(http.StatusBadRequest, \"invalid node pool %q: %v\", pool.Name, err)\n\t\t}\n\t\tif pool.IsBuiltIn() {\n\t\t\treturn structs.NewErrRPCCodedf(http.StatusBadRequest, \"modifying node pool %q is not allowed\", pool.Name)\n\t\t}\n\n\t\tpool.SetHash()","sourceCodeStart":177,"sourceCodeEnd":213,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/node_pool_endpoint.go#L177-L213","documentation":"Node pools require every server in the region to run at least minNodePoolsVersion. UpsertNodePools consults the peers cache (ServersMeetMinimumVersion with allowMissing=false) and refuses the upsert if any peer is older. This prevents mixed-version state where older servers cannot handle node pool objects.","triggerScenarios":"Calling NodePool.UpsertNodePools (API/CLI `nomad node pool upsert` or POST /v1/node/pools) while one or more servers in the region run a Nomad version older than minNodePoolsVersion (Nomad 1.1.0), or stale peers-cache data from a server that recently joined/left.","commonSituations":"Rolling cluster upgrades: operator applies node-pool config while a lagging server hasn't been upgraded yet; a server stuck on an old version; peers cache not yet refreshed after upgrade.","solutions":["Upgrade all servers in the region to at least minNodePoolsVersion (1.1.0) before retrying","Verify with `nomad server members` that every server reports the expected version and is alive","Retry after the peers cache refreshes / after the upgraded server rejoins consensus","Retry the node pool upsert once all members are current"],"exampleFix":"// before\n$ nomad node pool upsert prod.hcl   # one server on 1.0.x\n// after\n$ nomad server members               # confirm all servers >= 1.1.0\n$ nomad node pool upsert prod.hcl","handlingStrategy":"validation","validationCode":"members, _ := client.Agent().Members()\nfor _, m := range members.Members {\n    if version.Compare(version.Must(version.Parse(m.Tags[\"build\"]), nil), minNodePoolsVersion) < 0 {\n        return fmt.Errorf(\"server %s too old for node pools\", m.Name)\n    }\n}","typeGuard":null,"tryCatchPattern":"err := client.NodePools().Upsert(pool, wr)\nif err != nil && strings.Contains(err.Error(), \"all servers must be running version\") {\n    // defer config change until upgrade completes\n}","preventionTips":["Complete rolling upgrades before applying node-pool config","Check `nomad server members` versions in CI/deploy scripts","Gate infrastructure-as-code node-pool changes on cluster version","Remove stale pre-1.1.0 servers from the Raft config"],"tags":["nomad","node-pools","versioning","rolling-upgrade"],"backgroundTag":"cluster-version-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}