hashicorp/nomad · error

all servers should be running version

Error message

all servers should be running version %v or later to use dynamic host volumes

What it means

The HostVolume.Create RPC requires every server in the region to support dynamic host volumes (introduced behind a minimum version). The endpoint checks ServersMeetMinimumVersion against minVersionDynamicHostVolumes and fails with this message if any server is too old, refusing the create so the cluster state stays consistent.

Solutions

1. Run `nomad server members` and find servers below the required version.
2. Upgrade the outdated servers to the minimum version that supports dynamic host volumes.
3. After upgrading, wait for the peers cache to refresh (or restart the client connection) and retry the create.
4. If an old server is decommissioned but still listed, remove it from the cluster so the version check passes.

Example fix

# before: check fleet versions
nomad server members
# upgrade lagging server, then retry
nomad host volume create -name web-vol -host-path /srv/volumes/web

// check minimum version across the fleet before dynamic host volume create
for _, m := range members.Members {
    if m.Tags["build"] < "1.7.0" { // minVersionDynamicHostVolumes
        return fmt.Errorf("server %s too old for dynamic host volumes", m.Name)
    }
}

err := createHostVolume(req)
if err != nil && strings.Contains(err.Error(), "all servers should be running version") {
    // pause creates, complete upgrade, then retry
}

When it happens

Trigger: Calling `nomad host volume create` / the HostVolume Create RPC while at least one server in the region runs a Nomad version older than minVersionDynamicHostVolumes.

Common situations: Rolling upgrades that stalled mid-way, forgotten secondary servers left on old versions, or operators attempting dynamic host volumes before completing the upgrade.

Related errors

• all servers should be running version
• could not place volume
• error
• all servers must be running version
• All servers should be running version

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7feb8e21605f6e73. Report an issue: GitHub.

Appendix: source

Thrown at nomad/host_volume_endpoint.go:200

func (v *HostVolume) Create(args *structs.HostVolumeCreateRequest, reply *structs.HostVolumeCreateResponse) error {

	authErr := v.srv.Authenticate(v.ctx, args)
	if done, err := v.srv.forward("HostVolume.Create", args, args, reply); done {
		return err
	}
	v.srv.MeasureRPCRate("host_volume", structs.RateMetricWrite, args)
	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "host_volume", "create"}, time.Now())

	if !v.srv.peersCache.ServersMeetMinimumVersion(
		v.srv.Region(),
		minVersionDynamicHostVolumes,
		false,
	) {
		return fmt.Errorf(
			"all servers should be running version %v or later to use dynamic host volumes",
			minVersionDynamicHostVolumes,
		)
	}

	allowVolume := acl.NamespaceValidator(acl.NamespaceCapabilityHostVolumeCreate)
	aclObj, err := v.srv.ResolveACL(args)
	if err != nil {
		return err
	}

	if args.Volume == nil {
		return fmt.Errorf("missing volume definition")
	}

	vol := args.Volume
	if vol.Namespace == "" {
		vol.Namespace = args.RequestNamespace()

View on GitHub (pinned to 482b49bf1a)