hashicorp/nomad · error

all servers must be running version

Error message

all servers must be running version %v or later to delete node pools

What it means

DeleteNodePools enforces the same minimum-version gate as upsert: every server in the region must run minNodePoolsVersion or later before a node pool can be deleted. It uses the peers cache check and refuses the delete otherwise, preventing older servers from encountering node pool tombstones they cannot interpret.

Solutions

1. Upgrade all servers to at least minNodePoolsVersion before deleting pools
2. Confirm with `nomad server members` that no pre-1.1.0 servers remain
3. Wait for peers cache refresh or leader election to settle, then retry
4. Remove/replacement of any dead-but-listed old-version server in the Raft config

Example fix

// before
$ nomad node pool delete legacy      # mixed-version cluster
// after
$ nomad server members               # all >= 1.1.0
$ nomad node pool delete legacy

members, _ := client.Agent().Members()
for _, m := range members.Members {
    if version.Compare(version.Must(version.Parse(m.Tags["build"]), nil), minNodePoolsVersion) < 0 {
        return fmt.Errorf("cannot delete pools: server %s too old", m.Name)
    }
}

err := client.NodePools().Delete(names, wr)
if err != nil && strings.Contains(err.Error(), "all servers must be running version") {
    // postpone deletion until all servers are upgraded
}

When it happens

Trigger: Calling NodePool.DeleteNodePools (`nomad node pool delete` or DELETE /v1/node/pools) while any server in the region is older than minNodePoolsVersion (1.1.0).

Common situations: Deleting a pool during a partial rolling upgrade; an old canary server still in the Raft peer list; peers cache briefly stale after a server upgrade.

Related errors

• all servers must be running version
• all servers should be running version
• deleting node pool is not allowed
• error
• Feature "Node Pools Governance" is unlicensed

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e47395bd4d0ce2cb. Report an issue: GitHub.

Appendix: source

Thrown at nomad/node_pool_endpoint.go:260

	// Resolve ACL token and verify it has delete capability to all pools in
	// the request.
	aclObj, err := n.srv.ResolveACL(args)
	if err != nil {
		return err
	}
	for _, name := range args.Names {
		if !aclObj.AllowNodePoolOperation(name, acl.NodePoolCapabilityDelete) {
			return structs.ErrPermissionDenied
		}
	}

	// Only warn for expiration on delete because just parts of node pools are
	// licensed, so they are allowed to be deleted.
	_ = n.validateLicense(nil)

	if !n.srv.peersCache.ServersMeetMinimumVersion(n.srv.Region(), minNodePoolsVersion, true) {
		return fmt.Errorf("all servers must be running version %v or later to delete node pools", minNodePoolsVersion)
	}

	// Validate request.
	if len(args.Names) == 0 {
		return structs.NewErrRPCCodedf(http.StatusBadRequest, "must specify at least one node pool to delete")
	}
	if slices.Contains(args.Names, "") {
		return structs.NewErrRPCCodedf(http.StatusBadRequest, "node pool name is empty")
	}

	// Verify that the node pools we're deleting do not have nodes or
	// non-terminal jobs in this region or in any federated region.
	var mErr multierror.Error
	for _, name := range args.Names {
		regionsWithNonTerminal, regionsWithNodes, err := n.nodePoolRegionsInUse(args.AuthToken, name)
		if err != nil {
			_ = multierror.Append(&mErr, err)
		}

View on GitHub (pinned to 482b49bf1a)