hashicorp/nomad · error

all servers should be running version %v or later to use cli

Error message

all servers should be running version %v or later to use client intro tokens

What it means

Nomad's CreateClientIntroductionToken RPC can only issue client introduction tokens once every server in the local region is running at least the minVersionNodeIntro release. The server checks its peersCache via ServersMeetMinimumVersion and rejects the request with this message if any peer is older. It is a cluster-federation/upgrade-consistency guard, not a client bug.

Source

Thrown at nomad/acl_endpoint.go:3202

	}
	return j, nil
}

func (a *ACL) CreateClientIntroductionToken(
	args *structs.ACLCreateClientIntroductionTokenRequest,
	reply *structs.ACLCreateClientIntroductionTokenResponse) error {

	authErr := a.srv.Authenticate(a.ctx, args)

	if done, err := a.srv.forward(structs.ACLCreateClientIntroductionTokenRPCMethod, args, args, reply); done {
		return err
	}
	a.srv.MeasureRPCRate("acl", structs.RateMetricWrite, args)

	// This endpoint can only be used once all servers in the local region have
	// been upgraded to minVersionNodeIntro or greater.
	if !a.srv.peersCache.ServersMeetMinimumVersion(a.srv.Region(), minVersionNodeIntro, false) {
		return fmt.Errorf(
			"all servers should be running version %v or later to use client intro tokens",
			minVersionNodeIntro)
	}

	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{
		"nomad", "acl", "create_node_introduction_identity"}, time.Now())

	// Unlike the other ACL RPCs, this accepts node write permissions rather
	// than management. This allows cluster administrators to delegate node
	// introduction identity operations to other users who can bring their own
	// nodes to join the cluster.
	if aclObj, err := a.srv.ResolveACL(args); err != nil {
		return err
	} else if !aclObj.AllowNodeWrite() {
		return structs.ErrPermissionDenied

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all servers in the region to at least the required version (minVersionNodeIntro) and wait for the upgrade to finish
2. Verify with `nomad server members` / `nomad version` that every server reports the minimum version
3. Remove or restart any stale/outdated server from the region's peer list, then retry
4. If the upgrade is intentional and ongoing, retry the request after the rolling upgrade completes

Example fix

// before: mixing versions during rolling upgrade
// after: confirm all servers upgraded
$ nomad server members  # all servers >= minVersionNodeIntro
$ nomad acl token create ... # retry client intro token request

members, _ := agentClient.Agent().Members()
requiredVersion := "1.9.0" // minVersionNodeIntro
for _, m := range members.Members {
    if v, err := version.NewVersion(m.Tags["build"]); err != nil || v.LessThan(required) {
        return fmt.Errorf("server %s below %s, intro tokens unavailable", m.Name, requiredVersion)
    }
}

var versionSkewErr = regexp.MustCompile(`all servers should be running version`)
if versionSkewErr.MatchString(err.Error()) {
    // defer token issuance until the rolling upgrade completes
}

When it happens

Trigger: Calling the ClientIntroductionToken RPC (or agent APIs that use it, e.g. node introduction flows) while at least one server in the local region runs a Nomad version older than minVersionNodeIntro. The check happens after rate measurement and before auth error evaluation.

Common situations: Rolling upgrade still in progress; a lagging server failed to upgrade and was left in the peer list; mixed-version multi-region cluster where one region was not upgraded; server rejoined the gossip pool with old binary after a rollback.

Related errors

• All servers should be running version %v or later to use one
• all servers must be running version %v or later to delete ev
• all servers should be running version %v or later to use dyn
• no servers
• missing NodeID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/25866c259f2a77a4. Report an issue: GitHub.