hashicorp/nomad · critical

failed to set cpuset.mems on share cpuset partition: %w

Error message

failed to set cpuset.mems on share cpuset partition: %w

What it means

During cgroupslib.Init in CG1 mode, writing the mems value to /sys/fs/cgroup/cpuset/<nomad-parent>/share/cpuset.mems failed. The share partition must be pinned to valid memory nodes before task cgroups can be created under it. Failures are typically permission errors or EINVAL if the mems value is empty/invalid relative to the parent.

Source

Thrown at client/lib/cgroupslib/init.go:105

		if err := writeCG(cores, "cpuset", NomadCgroupParent, cpusetFile); err != nil {
			return fmt.Errorf("failed to write cores to nomad cpuset cgroup: %w", err)
		}

		//
		// share partition
		//

		if err := mkCG("cpuset", NomadCgroupParent, SharePartition()); err != nil {
			return fmt.Errorf("failed to create share cpuset partition: %w", err)
		}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, SharePartition(), cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, SharePartition(), memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on share cpuset partition: %w", err)
		}

		//
		// reserve partition
		//

		if err := mkCG("cpuset", NomadCgroupParent, ReservePartition()); err != nil {
			return fmt.Errorf("failed to create reserve cpuset partition: %w", err)
		}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, ReservePartition(), cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, ReservePartition(), memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on reserve cpuset partition: %w", err)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify /sys/fs/cgroup/cpuset/cpuset.mems at the root contains a valid mask (e.g. 0) before starting Nomad; if empty, assign nodes first.
2. Run the agent with sufficient privileges to write cgroupfs.
3. Ensure cgroupfs is mounted rw.
4. Confirm the detected mems value (root or pre-existing parent) is valid for this system's NUMA topology.

mems=$(cat /sys/fs/cgroup/cpuset/cpuset.mems 2>/dev/null)
if [ -z "$mems" ] || [ "$mems" = "" ]; then
  echo "root cpuset.mems empty; assign memory nodes before starting Nomad"
fi
cat /sys/fs/cgroup/cpuset/nomad/share/cpuset.mems 2>/dev/null || echo "share cpuset.mems missing"

if err := cgroupslib.Init(logger, cores); err != nil {
    if strings.Contains(err.Error(), "cpuset.mems on share") {
        return fmt.Errorf("invalid/undetected mems mask; verify root cpuset.mems and privileges: %w", err)
    }
    return err
}

When it happens

Trigger: os.WriteFile on the share partition's cpuset.mems with the detected memsSet fails during Init CG1.

Common situations: Root cpuset.mems was empty (cpuset controller newly enabled with no nodes assigned), so writing an empty/invalid value is rejected; non-root agent; container with read-only cgroupfs.

Related errors

• failed to write cores to nomad cpuset cgroup: %w
• failed to set cpuset.mems on reserve cpuset partition: %w
• failed to set cpuset.mems on nomad cpuset cgroup: %w
• failed to create share cpuset partition: %w
• failed to create reserve cpuset partition: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/299f8a038ec6c389. Report an issue: GitHub.