failed to set cpuset.mems on reserve cpuset partition: %w

Error message

failed to set cpuset.mems on reserve cpuset partition: %w

What it means

During cgroupslib.Init in CG1 mode, writing the mems value to /sys/fs/cgroup/cpuset/<nomad-parent>/reserve/cpuset.mems failed. The reserve partition needs a valid memory-node mask before reserved-core task cgroups are created beneath it. Failures mirror the share partition case: permission errors, read-only mounts, or an invalid/empty mems value.

Source

Thrown at client/lib/cgroupslib/init.go:121

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, SharePartition(), memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on share cpuset partition: %w", err)
		}

		//
		// reserve partition
		//

		if err := mkCG("cpuset", NomadCgroupParent, ReservePartition()); err != nil {
			return fmt.Errorf("failed to create reserve cpuset partition: %w", err)
		}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, ReservePartition(), cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, ReservePartition(), memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on reserve cpuset partition: %w", err)
		}

		log.Debug("nomad cpuset partitions initialized", "cores", cores)

	case CG2:
		// the cgroup controllers we need to activate at the root and on the nomad slice
		const activation = "+cpuset +cpu +io +memory +pids"

		// the name of the cgroup subtree interface file
		const subtreeFile = "cgroup.subtree_control"

		//
		// configuring root cgroup (/sys/fs/cgroup)
		//
		// clients with delegated cgroups typically won't be able to write to
		// the subtree file, but that's ok so long as the required controllers
		// are activated
		if !functionalCgroups2(subtreeFile) {

View on GitHub (pinned to 482b49bf1a)

Solutions

Check root cpuset.mems holds a valid mask and the memsSet value matches allowed nodes.
Run Nomad with privileges to write cgroupfs.
Ensure cgroupfs is mounted rw.
Delete a stale/incompletely configured nomad cpuset parent and let Init rebuild it.

Defensive patterns

Strategy: validation

Validate before calling

mems=$(cat /sys/fs/cgroup/cpuset/cpuset.mems 2>/dev/null)
[ -n "$mems" ] || echo "root cpuset.mems empty; fix before starting Nomad"
cat /sys/fs/cgroup/cpuset/nomad/reserve/cpuset.mems 2>/dev/null || echo "reserve cpuset.mems missing"

Try / catch

if err := cgroupslib.Init(logger, cores); err != nil {
    if strings.Contains(err.Error(), "cpuset.mems on reserve") {
        return fmt.Errorf("failed to pin mems on reserve partition; verify mems mask and privileges: %w", err)
    }
    return err
}

Prevention

Validate the root cpuset.mems mask before agent startup.
Run with sufficient cgroupfs write privileges.
Re-init cleanly if the nomad parent was configured out of band with stale mems.

When it happens

Trigger: os.WriteFile on the reserve partition's cpuset.mems with the detected memsSet fails during Init CG1 — the final cpuset write before Init logs success.

Common situations: Empty root cpuset.mems producing an invalid value; non-root agent; container with read-only cgroup mount; NUMA topology changes after an out-of-band parent mems value was cached.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c14cb127107ffa2c. Report an issue: GitHub.