All servers should be running version %v or later to use one

Error message

All servers should be running version %v or later to use one-time authentication tokens

What it means

UpsertOneTimeToken creates a one-time authentication token for a workload, a feature introduced at minOneTimeAuthenticationTokenVersion. Because RPCs are handled cluster-wide, every server in the local region must understand the feature; if any peer is older, the endpoint refuses the request with this error to avoid state that older servers cannot process.

Source

Thrown at nomad/acl_endpoint.go:1077

}

func (a *ACL) UpsertOneTimeToken(args *structs.OneTimeTokenUpsertRequest, reply *structs.OneTimeTokenUpsertResponse) error {
	if !a.srv.config.ACLEnabled {
		return aclDisabled
	}
	if done, err := a.srv.forward(
		"ACL.UpsertOneTimeToken", args, args, reply); done {
		return err
	}
	defer metrics.MeasureSince(
		[]string{"nomad", "acl", "upsert_one_time_token"}, time.Now())

	if !a.srv.peersCache.ServersMeetMinimumVersion(
		a.srv.Region(),
		minOneTimeAuthenticationTokenVersion,
		false,
	) {
		return fmt.Errorf(
			"All servers should be running version %v or later to use one-time authentication tokens",
			minOneTimeAuthenticationTokenVersion,
		)
	}

	// Snapshot the state
	state, err := a.srv.State().Snapshot()
	if err != nil {
		return err
	}

	// Look up the token; there's no capability check as you can only
	// request a OTT for your own ACL token
	aclToken, err := state.ACLTokenBySecretID(nil, args.AuthToken)
	if err != nil {
		return err
	}
	if aclToken == nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

Upgrade all servers in the region to at least minOneTimeAuthenticationTokenVersion, then retry
Check server versions with `nomad server members` and identify the lagging server
Complete or restart a stalled rolling upgrade (fix the server that didn't restart on the new binary)
Temporarily avoid one-time tokens (use standard ACL tokens) until the cluster is homogeneous

Defensive patterns

Strategy: retry

Validate before calling

members, _ := client.Agent().Members()
for _, m := range members.Members {
    if !serverMeetsMinVersion(m.Tags["build"], minVersion) {
        return fmt.Errorf("server %s at %s too old for one-time tokens", m.Name, m.Tags["build"])
    }
}

Try / catch

resp, err := acl.UpsertOneTimeToken(req)
if err != nil && strings.Contains(err.Error(), "All servers should be running version") {
    // wait for rolling upgrade to finish, retry with backoff
    time.AfterFunc(upgradeWait, func() { retryUpsert(req) })
    return
}

Prevention

Finish rolling upgrades before enabling one-time token features
Monitor server build versions and alert on skew
Pin cluster upgrades so no server lags the minimum feature version
Gate feature usage (one-time tokens) on a version check at deploy time

When it happens

Trigger: Invoking the ACL OneTimeToken Upsert RPC while at least one server in the region (excluding failing/left nodes per the don't-recheck flag) reports a version below minOneTimeAuthenticationTokenVersion via peersCache.ServersMeetMinimumVersion.

Common situations: Rolling upgrade in progress with mixed server versions; a node that failed to upgrade stuck on an old binary; newly joined server still on a prior release.

Understand the failure class

Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/009aa852f807974b. Report an issue: GitHub.