All servers should be running version

Error message

All servers should be running version %v or later to use one-time authentication tokens

What it means

UpsertOneTimeToken creates a one-time authentication token for a workload, a feature introduced at minOneTimeAuthenticationTokenVersion. Because RPCs are handled cluster-wide, every server in the local region must understand the feature; if any peer is older, the endpoint refuses the request with this error to avoid state that older servers cannot process.

Solutions

Upgrade all servers in the region to at least minOneTimeAuthenticationTokenVersion, then retry
Check server versions with `nomad server members` and identify the lagging server
Complete or restart a stalled rolling upgrade (fix the server that didn't restart on the new binary)
Temporarily avoid one-time tokens (use standard ACL tokens) until the cluster is homogeneous

Defensive patterns

Strategy: retry

Validate before calling

members, _ := client.Agent().Members()
for _, m := range members.Members {
    if !serverMeetsMinVersion(m.Tags["build"], minVersion) {
        return fmt.Errorf("server %s at %s too old for one-time tokens", m.Name, m.Tags["build"])
    }
}

Try / catch

resp, err := acl.UpsertOneTimeToken(req)
if err != nil && strings.Contains(err.Error(), "All servers should be running version") {
    // wait for rolling upgrade to finish, retry with backoff
    time.AfterFunc(upgradeWait, func() { retryUpsert(req) })
    return
}

Prevention

Finish rolling upgrades before enabling one-time token features
Monitor server build versions and alert on skew
Pin cluster upgrades so no server lags the minimum feature version
Gate feature usage (one-time tokens) on a version check at deploy time

When it happens

Trigger: Invoking the ACL OneTimeToken Upsert RPC while at least one server in the region (excluding failing/left nodes per the don't-recheck flag) reports a version below minOneTimeAuthenticationTokenVersion via peersCache.ServersMeetMinimumVersion.

Common situations: Rolling upgrade in progress with mixed server versions; a node that failed to upgrade stuck on an old binary; newly joined server still on a prior release.

Understand the failure class

Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/009aa852f807974b. Report an issue: GitHub.

Appendix: source

Thrown at nomad/acl_endpoint.go:1077

}

func (a *ACL) UpsertOneTimeToken(args *structs.OneTimeTokenUpsertRequest, reply *structs.OneTimeTokenUpsertResponse) error {
	if !a.srv.config.ACLEnabled {
		return aclDisabled
	}
	if done, err := a.srv.forward(
		"ACL.UpsertOneTimeToken", args, args, reply); done {
		return err
	}
	defer metrics.MeasureSince(
		[]string{"nomad", "acl", "upsert_one_time_token"}, time.Now())

	if !a.srv.peersCache.ServersMeetMinimumVersion(
		a.srv.Region(),
		minOneTimeAuthenticationTokenVersion,
		false,
	) {
		return fmt.Errorf(
			"All servers should be running version %v or later to use one-time authentication tokens",
			minOneTimeAuthenticationTokenVersion,
		)
	}

	// Snapshot the state
	state, err := a.srv.State().Snapshot()
	if err != nil {
		return err
	}

	// Look up the token; there's no capability check as you can only
	// request a OTT for your own ACL token
	aclToken, err := state.ACLTokenBySecretID(nil, args.AuthToken)
	if err != nil {
		return err
	}
	if aclToken == nil {

View on GitHub (pinned to 482b49bf1a)