hashicorp/nomad · error

all servers must be running version %v or later to apply var

Error message

all servers must be running version %v or later to apply variables

What it means

Variables are secured with a keyring introduced in a minimum Nomad server version. Before doing any work, Apply checks via peersCache that ALL servers in the region meet the minVersionKeyring version; if any server is older, Apply refuses with this error. This prevents writing encrypted variables that older servers could not handle.

Source

Thrown at nomad/variables_endpoint.go:90

	defer metrics.MeasureSince([]string{
		"nomad", "variables", "apply", string(args.Op)}, time.Now())
	// TODO: Add metrics for acquire and release if the operation is lock related

	if args.Var == nil {
		return fmt.Errorf("variable must not be nil")
	}

	// Check if the Namespace is explicitly set on the variable. If
	// not, use the RequestNamespace
	targetNS := args.Var.Namespace
	if targetNS == "" {
		targetNS = args.RequestNamespace()
		args.Var.Namespace = targetNS
	}

	if !sv.srv.peersCache.ServersMeetMinimumVersion(sv.srv.Region(), minVersionKeyring, true) {
		return fmt.Errorf("all servers must be running version %v or later to apply variables", minVersionKeyring)
	}

	// Perform the ACL resolution.
	aclObj, err := sv.srv.ResolveACL(args)
	if err != nil {
		return err
	}
	err = hasOperationPermissions(aclObj, args.Var.Namespace, args.Var.Path, args.Op)
	if err != nil {
		return err
	}

	err = canonicalizeAndValidate(args)
	if err != nil {
		return structs.NewErrRPCCoded(http.StatusBadRequest, err.Error())
	}

	var ev *structs.VariableEncrypted

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all servers to at least minVersionKeyring (check `nomad server members` for versions)
2. Remove or drain the outdated server from the cluster
3. Retry the variable apply once every server reports the minimum version
4. Restart any server still running an old binary after upgrade

Example fix

// before
# mixed versions 1.4.x / 1.7.x
nomad var put app/config k=v  # blocked

// after
# upgrade all servers, then verify:
nomad server members  # all >= min version
nomad var put app/config k=v

members, _ := agent.Members()
for _, m := range members.Members {
    if !serverMeetsKeyringVersion(m.Tags["version"]) {
        return fmt.Errorf("server %s at %s is below keyring minimum version", m.Name, m.Addr)
    }
}

_, err := client.Variables().Apply(req, nil)
if err != nil && strings.Contains(err.Error(), "all servers must be running version") {
    // schedule/await server upgrades, then retry
}

When it happens

Trigger: Applying any variable while at least one server in the region runs a Nomad version older than minVersionKeyring (mixed-version cluster during rolling upgrade, or a forgotten lagging server).

Common situations: Rolling upgrades where one server was never updated; a re-joined old server binary; enterprise/oss mismatch; users of a cluster with heterogeneous server versions trying `nomad var put`.

Related errors

• all servers must be running version %v or later to delete ev
• LockNoPathErr
• variable not found
• root key not found
• root key in use, cannot delete

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/045776f2b3347b21. Report an issue: GitHub.