{"record":{"id":"7feb8e21605f6e73","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later-7feb8e","errorCode":null,"errorMessage":"all servers should be running version %v or later to use dynamic host volumes","messagePattern":"all servers should be running version (.+?) or later to use dynamic host volumes","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/host_volume_endpoint.go","lineNumber":200,"sourceCode":"\nfunc (v *HostVolume) Create(args *structs.HostVolumeCreateRequest, reply *structs.HostVolumeCreateResponse) error {\n\n\tauthErr := v.srv.Authenticate(v.ctx, args)\n\tif done, err := v.srv.forward(\"HostVolume.Create\", args, args, reply); done {\n\t\treturn err\n\t}\n\tv.srv.MeasureRPCRate(\"host_volume\", structs.RateMetricWrite, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"host_volume\", \"create\"}, time.Now())\n\n\tif !v.srv.peersCache.ServersMeetMinimumVersion(\n\t\tv.srv.Region(),\n\t\tminVersionDynamicHostVolumes,\n\t\tfalse,\n\t) {\n\t\treturn fmt.Errorf(\n\t\t\t\"all servers should be running version %v or later to use dynamic host volumes\",\n\t\t\tminVersionDynamicHostVolumes,\n\t\t)\n\t}\n\n\tallowVolume := acl.NamespaceValidator(acl.NamespaceCapabilityHostVolumeCreate)\n\taclObj, err := v.srv.ResolveACL(args)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tif args.Volume == nil {\n\t\treturn fmt.Errorf(\"missing volume definition\")\n\t}\n\n\tvol := args.Volume\n\tif vol.Namespace == \"\" {\n\t\tvol.Namespace = args.RequestNamespace()","sourceCodeStart":182,"sourceCodeEnd":218,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/host_volume_endpoint.go#L182-L218","documentation":"The HostVolume.Create RPC requires every server in the region to support dynamic host volumes (introduced behind a minimum version). The endpoint checks ServersMeetMinimumVersion against minVersionDynamicHostVolumes and fails with this message if any server is too old, refusing the create so the cluster state stays consistent.","triggerScenarios":"Calling `nomad host volume create` / the HostVolume Create RPC while at least one server in the region runs a Nomad version older than minVersionDynamicHostVolumes.","commonSituations":"Rolling upgrades that stalled mid-way, forgotten secondary servers left on old versions, or operators attempting dynamic host volumes before completing the upgrade.","solutions":["Run `nomad server members` and find servers below the required version.","Upgrade the outdated servers to the minimum version that supports dynamic host volumes.","After upgrading, wait for the peers cache to refresh (or restart the client connection) and retry the create.","If an old server is decommissioned but still listed, remove it from the cluster so the version check passes."],"exampleFix":"# before: check fleet versions\nnomad server members\n# upgrade lagging server, then retry\nnomad host volume create -name web-vol -host-path /srv/volumes/web","handlingStrategy":"validation","validationCode":"// check minimum version across the fleet before dynamic host volume create\nfor _, m := range members.Members {\n    if m.Tags[\"build\"] < \"1.7.0\" { // minVersionDynamicHostVolumes\n        return fmt.Errorf(\"server %s too old for dynamic host volumes\", m.Name)\n    }\n}","typeGuard":null,"tryCatchPattern":"err := createHostVolume(req)\nif err != nil && strings.Contains(err.Error(), \"all servers should be running version\") {\n    // pause creates, complete upgrade, then retry\n}","preventionTips":["Verify `nomad server members` shows all servers at the required version","Gate dynamic host volume usage behind cluster-upgrade completion checks","Remove decommissioned old servers from the raft pool"],"tags":["nomad","host-volumes","version-skew","rpc"],"backgroundTag":"version-skew-unsupported-operation","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}