{"record":{"id":"045776f2b3347b21","repo":"hashicorp/nomad","slug":"all-servers-must-be-running-version-v-or-later-to-045776","errorCode":null,"errorMessage":"all servers must be running version %v or later to apply variables","messagePattern":"all servers must be running version (.+?) or later to apply variables","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/variables_endpoint.go","lineNumber":90,"sourceCode":"\n\tdefer metrics.MeasureSince([]string{\n\t\t\"nomad\", \"variables\", \"apply\", string(args.Op)}, time.Now())\n\t// TODO: Add metrics for acquire and release if the operation is lock related\n\n\tif args.Var == nil {\n\t\treturn fmt.Errorf(\"variable must not be nil\")\n\t}\n\n\t// Check if the Namespace is explicitly set on the variable. If\n\t// not, use the RequestNamespace\n\ttargetNS := args.Var.Namespace\n\tif targetNS == \"\" {\n\t\ttargetNS = args.RequestNamespace()\n\t\targs.Var.Namespace = targetNS\n\t}\n\n\tif !sv.srv.peersCache.ServersMeetMinimumVersion(sv.srv.Region(), minVersionKeyring, true) {\n\t\treturn fmt.Errorf(\"all servers must be running version %v or later to apply variables\", minVersionKeyring)\n\t}\n\n\t// Perform the ACL resolution.\n\taclObj, err := sv.srv.ResolveACL(args)\n\tif err != nil {\n\t\treturn err\n\t}\n\terr = hasOperationPermissions(aclObj, args.Var.Namespace, args.Var.Path, args.Op)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = canonicalizeAndValidate(args)\n\tif err != nil {\n\t\treturn structs.NewErrRPCCoded(http.StatusBadRequest, err.Error())\n\t}\n\n\tvar ev *structs.VariableEncrypted","sourceCodeStart":72,"sourceCodeEnd":108,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/variables_endpoint.go#L72-L108","documentation":"Variables are secured with a keyring introduced in a minimum Nomad server version. Before doing any work, Apply checks via peersCache that ALL servers in the region meet the minVersionKeyring version; if any server is older, Apply refuses with this error. This prevents writing encrypted variables that older servers could not handle.","triggerScenarios":"Applying any variable while at least one server in the region runs a Nomad version older than minVersionKeyring (mixed-version cluster during rolling upgrade, or a forgotten lagging server).","commonSituations":"Rolling upgrades where one server was never updated; a re-joined old server binary; enterprise/oss mismatch; users of a cluster with heterogeneous server versions trying `nomad var put`.","solutions":["Upgrade all servers to at least minVersionKeyring (check `nomad server members` for versions)","Remove or drain the outdated server from the cluster","Retry the variable apply once every server reports the minimum version","Restart any server still running an old binary after upgrade"],"exampleFix":"// before\n# mixed versions 1.4.x / 1.7.x\nnomad var put app/config k=v  # blocked\n\n// after\n# upgrade all servers, then verify:\nnomad server members  # all >= min version\nnomad var put app/config k=v","handlingStrategy":"validation","validationCode":"members, _ := agent.Members()\nfor _, m := range members.Members {\n    if !serverMeetsKeyringVersion(m.Tags[\"version\"]) {\n        return fmt.Errorf(\"server %s at %s is below keyring minimum version\", m.Name, m.Addr)\n    }\n}","typeGuard":null,"tryCatchPattern":"_, err := client.Variables().Apply(req, nil)\nif err != nil && strings.Contains(err.Error(), \"all servers must be running version\") {\n    // schedule/await server upgrades, then retry\n}","preventionTips":["Complete rolling upgrades fully; verify `nomad server members` versions","Remove decommissioned/old servers from the cluster","Gate variable usage on a post-upgrade version check"],"tags":["nomad","variables","version-compatibility","cluster-upgrade"],"backgroundTag":"server-version-too-old","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}