hashicorp/nomad · error

all servers should be running version %v or later to use JWT

Error message

all servers should be running version %v or later to use JWT ACL auth methods

What it means

Thrown by the Login RPC when the cluster's servers do not all meet minACLJWTAuthMethodVersion — JWT-based ACL auth methods require every server in every region to run at least that Nomad version. It is an RPC preflight guard so the mixed-version cluster never partially processes JWT logins.

Source

Thrown at nomad/acl_endpoint.go:2941

	// Perform the initial forwarding within the region. This ensures we
	// respect stale queries.
	if done, err := a.srv.forward(structs.ACLLoginRPCMethod, args, args, reply); done {
		return err
	}

	// Measure the login endpoint performance.
	defer metrics.MeasureSince([]string{"nomad", "acl", "login"}, time.Now())

	// This endpoint can only be used once all servers in all federated regions
	// have been upgraded to minACLJWTAuthMethodVersion or greater, since JWT Auth
	// method was introduced then.
	if !a.srv.peersCache.ServersMeetMinimumVersion(
		peers.AllRegions,
		minACLJWTAuthMethodVersion,
		false,
	) {
		return fmt.Errorf(
			"all servers should be running version %v or later to use JWT ACL auth methods",
			minACLJWTAuthMethodVersion,
		)
	}

	// Validate the request arguments to ensure it contains all the data it
	// needs.
	if err := args.Validate(); err != nil {
		return structs.NewErrRPCCodedf(http.StatusBadRequest, "invalid login request: %v", err)
	}

	// Grab a snapshot of the state, so we can query it safely.
	stateSnapshot, err := a.srv.fsm.State().Snapshot()
	if err != nil {
		return err
	}

	// Lookup the auth method from state, so we have the entire object

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all Nomad servers to at least minACLJWTAuthMethodVersion and wait for the rolling upgrade to finish.
2. Check nomad server members / raft peers for any node still running an older version; remove or upgrade it.
3. Retry Login after the cluster converges (autoupdate / scheduler version reporting can lag briefly).
4. Use a non-JWT auth method (e.g. an auth method type supported by the older cluster) as a temporary workaround.

Example fix

// before: login while a 1.3.x server is still in the cluster
nomad acl login -method=jwt
// after: upgrade stragglers, then
nomad version  # ensure every server >= minACLJWTAuthMethodVersion
nomad acl login -method=jwt

// check server versions before attempting JWT login
out, _ := exec.Command("nomad", "server", "members").Output()
for _, m := range parseMembers(out) {
  if versionLessThan(m.Version, "1.4.0") { return fmt.Errorf("server %s too old", m.Name) }
}

err := client.ACL().Login(ctx, req)
if err != nil && strings.Contains(err.Error(), "version") {
    time.Sleep(retryInterval) // retry after rolling upgrade converges
}

When it happens

Trigger: A Login request with a JWT-type auth method arrives while peersCache.ServersMeetMinimumVersion(peers.AllRegions, minACLJWTAuthMethodVersion, false) reports at least one server below the required version.

Common situations: Rolling upgrade in progress, a stale/old server left in the region, or using JWT auth methods against an older cluster (e.g. pre-1.4.x servers) after a client was upgraded.

Related errors

• JWT auth method requires either OIDCDiscoveryURL, or JWKS UR
• missing login token
• Node does not support RPC; requires 0.8 or later
• invalid JWT issuer: %v
• node identity claims missing node pool

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6a634cd8abd8a558. Report an issue: GitHub.