{"record":{"id":"e47395bd4d0ce2cb","repo":"hashicorp/nomad","slug":"all-servers-must-be-running-version-v-or-later-to-e47395","errorCode":null,"errorMessage":"all servers must be running version %v or later to delete node pools","messagePattern":"all servers must be running version (.+?) or later to delete node pools","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/node_pool_endpoint.go","lineNumber":260,"sourceCode":"\n\t// Resolve ACL token and verify it has delete capability to all pools in\n\t// the request.\n\taclObj, err := n.srv.ResolveACL(args)\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, name := range args.Names {\n\t\tif !aclObj.AllowNodePoolOperation(name, acl.NodePoolCapabilityDelete) {\n\t\t\treturn structs.ErrPermissionDenied\n\t\t}\n\t}\n\n\t// Only warn for expiration on delete because just parts of node pools are\n\t// licensed, so they are allowed to be deleted.\n\t_ = n.validateLicense(nil)\n\n\tif !n.srv.peersCache.ServersMeetMinimumVersion(n.srv.Region(), minNodePoolsVersion, true) {\n\t\treturn fmt.Errorf(\"all servers must be running version %v or later to delete node pools\", minNodePoolsVersion)\n\t}\n\n\t// Validate request.\n\tif len(args.Names) == 0 {\n\t\treturn structs.NewErrRPCCodedf(http.StatusBadRequest, \"must specify at least one node pool to delete\")\n\t}\n\tif slices.Contains(args.Names, \"\") {\n\t\treturn structs.NewErrRPCCodedf(http.StatusBadRequest, \"node pool name is empty\")\n\t}\n\n\t// Verify that the node pools we're deleting do not have nodes or\n\t// non-terminal jobs in this region or in any federated region.\n\tvar mErr multierror.Error\n\tfor _, name := range args.Names {\n\t\tregionsWithNonTerminal, regionsWithNodes, err := n.nodePoolRegionsInUse(args.AuthToken, name)\n\t\tif err != nil {\n\t\t\t_ = multierror.Append(&mErr, err)\n\t\t}","sourceCodeStart":242,"sourceCodeEnd":278,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/node_pool_endpoint.go#L242-L278","documentation":"DeleteNodePools enforces the same minimum-version gate as upsert: every server in the region must run minNodePoolsVersion or later before a node pool can be deleted. It uses the peers cache check and refuses the delete otherwise, preventing older servers from encountering node pool tombstones they cannot interpret.","triggerScenarios":"Calling NodePool.DeleteNodePools (`nomad node pool delete` or DELETE /v1/node/pools) while any server in the region is older than minNodePoolsVersion (1.1.0).","commonSituations":"Deleting a pool during a partial rolling upgrade; an old canary server still in the Raft peer list; peers cache briefly stale after a server upgrade.","solutions":["Upgrade all servers to at least minNodePoolsVersion before deleting pools","Confirm with `nomad server members` that no pre-1.1.0 servers remain","Wait for peers cache refresh or leader election to settle, then retry","Remove/replacement of any dead-but-listed old-version server in the Raft config"],"exampleFix":"// before\n$ nomad node pool delete legacy      # mixed-version cluster\n// after\n$ nomad server members               # all >= 1.1.0\n$ nomad node pool delete legacy","handlingStrategy":"validation","validationCode":"members, _ := client.Agent().Members()\nfor _, m := range members.Members {\n    if version.Compare(version.Must(version.Parse(m.Tags[\"build\"]), nil), minNodePoolsVersion) < 0 {\n        return fmt.Errorf(\"cannot delete pools: server %s too old\", m.Name)\n    }\n}","typeGuard":null,"tryCatchPattern":"err := client.NodePools().Delete(names, wr)\nif err != nil && strings.Contains(err.Error(), \"all servers must be running version\") {\n    // postpone deletion until all servers are upgraded\n}","preventionTips":["Schedule pool deletions after upgrade windows close","Verify all member versions before destructive operations","Keep canary servers on the same version as the cluster","Monitor peers cache/leader stability post-upgrade"],"tags":["nomad","node-pools","versioning","rolling-upgrade"],"backgroundTag":"cluster-version-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}