all servers should be running version

Error message

all servers should be running version %v or later to use JWT ACL auth methods

What it means

Thrown by the Login RPC when the cluster's servers do not all meet minACLJWTAuthMethodVersion — JWT-based ACL auth methods require every server in every region to run at least that Nomad version. It is an RPC preflight guard so the mixed-version cluster never partially processes JWT logins.

Solutions

Upgrade all Nomad servers to at least minACLJWTAuthMethodVersion and wait for the rolling upgrade to finish.
Check nomad server members / raft peers for any node still running an older version; remove or upgrade it.
Retry Login after the cluster converges (autoupdate / scheduler version reporting can lag briefly).
Use a non-JWT auth method (e.g. an auth method type supported by the older cluster) as a temporary workaround.

Example fix

// before: login while a 1.3.x server is still in the cluster
nomad acl login -method=jwt
// after: upgrade stragglers, then
nomad version  # ensure every server >= minACLJWTAuthMethodVersion
nomad acl login -method=jwt

Defensive patterns

Strategy: retry

Validate before calling

// check server versions before attempting JWT login
out, _ := exec.Command("nomad", "server", "members").Output()
for _, m := range parseMembers(out) {
  if versionLessThan(m.Version, "1.4.0") { return fmt.Errorf("server %s too old", m.Name) }
}

Try / catch

err := client.ACL().Login(ctx, req)
if err != nil && strings.Contains(err.Error(), "version") {
    time.Sleep(retryInterval) // retry after rolling upgrade converges
}

Prevention

Complete rolling upgrades before enabling JWT auth methods.
Monitor nomad server members for stragglers on old versions.
Pin client tooling versions in sync with server minimum versions.

When it happens

Trigger: A Login request with a JWT-type auth method arrives while peersCache.ServersMeetMinimumVersion(peers.AllRegions, minACLJWTAuthMethodVersion, false) reports at least one server below the required version.

Common situations: Rolling upgrade in progress, a stale/old server left in the region, or using JWT auth methods against an older cluster (e.g. pre-1.4.x servers) after a client was upgraded.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6a634cd8abd8a558. Report an issue: GitHub.

Appendix: source

Thrown at nomad/acl_endpoint.go:2941

	// Perform the initial forwarding within the region. This ensures we
	// respect stale queries.
	if done, err := a.srv.forward(structs.ACLLoginRPCMethod, args, args, reply); done {
		return err
	}

	// Measure the login endpoint performance.
	defer metrics.MeasureSince([]string{"nomad", "acl", "login"}, time.Now())

	// This endpoint can only be used once all servers in all federated regions
	// have been upgraded to minACLJWTAuthMethodVersion or greater, since JWT Auth
	// method was introduced then.
	if !a.srv.peersCache.ServersMeetMinimumVersion(
		peers.AllRegions,
		minACLJWTAuthMethodVersion,
		false,
	) {
		return fmt.Errorf(
			"all servers should be running version %v or later to use JWT ACL auth methods",
			minACLJWTAuthMethodVersion,
		)
	}

	// Validate the request arguments to ensure it contains all the data it
	// needs.
	if err := args.Validate(); err != nil {
		return structs.NewErrRPCCodedf(http.StatusBadRequest, "invalid login request: %v", err)
	}

	// Grab a snapshot of the state, so we can query it safely.
	stateSnapshot, err := a.srv.fsm.State().Snapshot()
	if err != nil {
		return err
	}

	// Lookup the auth method from state, so we have the entire object

View on GitHub (pinned to 482b49bf1a)