What causes "failed to validate volume %s, err: %w" in hashicorp/nomad?

Read the wrapped os error (no such file / permission denied) to see which case applies

failed to validate volume %s, err: %w

Error message

failed to validate volume %s, err: %w

What it means

During client setup, every configured host volume's Path is checked with os.Stat before being registered on the node. If the path does not exist or is inaccessible, setup fails with this error wrapping the os.Stat error. This catches misconfigured volumes early instead of failing tasks later.

Source

Thrown at client/client.go:1665

				Topology: &numalib.Topology{},
			}
		}
	}
	if node.ReservedResources == nil {
		node.ReservedResources = &structs.NodeReservedResources{}
	}
	if node.Datacenter == "" {
		node.Datacenter = "dc1"
	}
	if node.Name == "" {
		node.Name, _ = os.Hostname()
	}
	node.CgroupParent = newConfig.CgroupParent
	if node.HostVolumes == nil {
		node.HostVolumes = make(map[string]*structs.ClientHostVolumeConfig, len(newConfig.HostVolumes))
		for k, v := range newConfig.HostVolumes {
			if _, err := os.Stat(v.Path); err != nil {
				return fmt.Errorf("failed to validate volume %s, err: %w", v.Name, err)
			}
			node.HostVolumes[k] = v.Copy()
		}
	}
	node.GCVolumesOnNodeGC = newConfig.GCVolumesOnNodeGC

	if node.HostNetworks == nil {
		if l := len(newConfig.HostNetworks); l != 0 {
			node.HostNetworks = make(map[string]*structs.ClientHostNetworkConfig, l)
			for k, v := range newConfig.HostNetworks {
				node.HostNetworks[k] = v.Copy()
			}
		}
	}

	if node.Name == "" {
		node.Name = node.ID
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the wrapped os error (no such file / permission denied) to see which case applies
Create the directory on the host (e.g. sudo mkdir -p /srv/nomad-volumes/data) or correct the path in the client host_volume config
Fix mount/permission issues (mount the volume, chown/chmod) and restart the Nomad client
Remove or comment out the host_volume entry if it is not used on this node

Example fix

// before
client {
  host_volume "data" {
    path = "/mnt/missing/nomad-data"
  }
}
// after
sudo mkdir -p /mnt/nomad/nomad-data
// then
client {
  host_volume "data" {
    path = "/mnt/nomad/nomad-data"
  }
}

Defensive patterns

Strategy: validation

Validate before calling

for _, v := range cfg.Client.HostVolumes {
	if _, err := os.Stat(v.Path); err != nil {
		return fmt.Errorf("host volume %q path %q invalid before agent start: %w", v.Name, v.Path, err)
	}
}

Try / catch

if err := validateHostVolumes(cfg); err != nil {
	logger.Warn("skipping invalid host volume", "err", err)
	// remove entry and reload config instead of failing the agent
}

Prevention

Provision volume directories with config management before the Nomad unit starts
Use systemd unit ordering (RequiresMountsFor=/path) for mounted volumes
Validate paths when templating configs for multiple machines
Bind-mount host paths into containers running Nomad

When it happens

Trigger: A host_volume block in the client config declares a host path that does not exist on disk, cannot be traversed (permission denied on a parent dir), or points at a broken symlink, so os.Stat(v.Path) fails.

Common situations: Config copied from another machine; volume path on a mount that is not mounted yet at agent start; typo in the path; running Nomad in a container without the host path bind-mounted.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6f1b2360903bcc8a. Report an issue: GitHub.