hashicorp/nomad · error

all servers must be running version %v or later to upsert no

Error message

all servers must be running version %v or later to upsert node pools

What it means

Node pools require every server in the region to run at least minNodePoolsVersion. UpsertNodePools consults the peers cache (ServersMeetMinimumVersion with allowMissing=false) and refuses the upsert if any peer is older. This prevents mixed-version state where older servers cannot handle node pool objects.

Source

Thrown at nomad/node_pool_endpoint.go:195

	// request.
	aclObj, err := n.srv.ResolveACL(args)
	if err != nil {
		return err
	}
	for _, pool := range args.NodePools {
		if !aclObj.AllowNodePoolOperation(pool.Name, acl.NodePoolCapabilityWrite) {
			return structs.ErrPermissionDenied
		}

		// Strict enforcement for write requests.
		// If not licensed then requests will be denied.
		if err := n.validateLicense(pool); err != nil {
			return err
		}
	}

	if !n.srv.peersCache.ServersMeetMinimumVersion(n.srv.Region(), minNodePoolsVersion, true) {
		return fmt.Errorf("all servers must be running version %v or later to upsert node pools", minNodePoolsVersion)
	}

	// Validate request.
	if len(args.NodePools) == 0 {
		return structs.NewErrRPCCodedf(http.StatusBadRequest, "must specify at least one node pool")
	}
	for _, pool := range args.NodePools {

		pool.Canonicalize()

		if err := pool.Validate(); err != nil {
			return structs.NewErrRPCCodedf(http.StatusBadRequest, "invalid node pool %q: %v", pool.Name, err)
		}
		if pool.IsBuiltIn() {
			return structs.NewErrRPCCodedf(http.StatusBadRequest, "modifying node pool %q is not allowed", pool.Name)
		}

		pool.SetHash()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all servers in the region to at least minNodePoolsVersion (1.1.0) before retrying
2. Verify with `nomad server members` that every server reports the expected version and is alive
3. Retry after the peers cache refreshes / after the upgraded server rejoins consensus
4. Retry the node pool upsert once all members are current

Example fix

// before
$ nomad node pool upsert prod.hcl   # one server on 1.0.x
// after
$ nomad server members               # confirm all servers >= 1.1.0
$ nomad node pool upsert prod.hcl

members, _ := client.Agent().Members()
for _, m := range members.Members {
    if version.Compare(version.Must(version.Parse(m.Tags["build"]), nil), minNodePoolsVersion) < 0 {
        return fmt.Errorf("server %s too old for node pools", m.Name)
    }
}

err := client.NodePools().Upsert(pool, wr)
if err != nil && strings.Contains(err.Error(), "all servers must be running version") {
    // defer config change until upgrade completes
}

When it happens

Trigger: Calling NodePool.UpsertNodePools (API/CLI `nomad node pool upsert` or POST /v1/node/pools) while one or more servers in the region run a Nomad version older than minNodePoolsVersion (Nomad 1.1.0), or stale peers-cache data from a server that recently joined/left.

Common situations: Rolling cluster upgrades: operator applies node-pool config while a lagging server hasn't been upgraded yet; a server stuck on an old version; peers cache not yet refreshed after upgrade.

Related errors

• all servers must be running version %v or later to delete no
• missing node pool name
• missing node pool
• Feature "Node Pools Governance" is unlicensed
• unsupported minimum common raft protocol version

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/01a52896ddf7de26. Report an issue: GitHub.