hashicorp/nomad · critical

failed to set cpuset.mems on nomad cpuset cgroup: %w

Error message

failed to set cpuset.mems on nomad cpuset cgroup: %w

What it means

During cgroupslib.Init in CG1 (cgroup v1) mode, writing the value "0" to /sys/fs/cgroup/cpuset/<nomad-parent>/cgroup.clone_children failed. Nomad sets clone_children=0 on its parent cpuset cgroup so that child cgroups do not automatically inherit cpus/mems and instead get explicit values. The write is done via os.WriteFile, so the wrapped error is typically EACCES (not root / no CAP_SYS_ADMIN) or ENOENT (cgroup controller not mounted).

Source

Thrown at client/lib/cgroupslib/init.go:85

		//
		// the tree is lopsided - tasks making use of reserved cpu cores get
		// their own cgroup with a static cpuset.cpus value. other tasks are
		// placed in the single share cgroup and share its dynamic cpuset.cpus
		// value
		//
		// e.g.,
		//  root/cpuset/nomad/
		//    share/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//    reserve/
		//      abc123.task/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//      def456.task/{cgroup.procs, cpuset.cpus, cpuset.mems}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(cores, "cpuset", NomadCgroupParent, cpusetFile); err != nil {
			return fmt.Errorf("failed to write cores to nomad cpuset cgroup: %w", err)
		}

		//
		// share partition
		//

		if err := mkCG("cpuset", NomadCgroupParent, SharePartition()); err != nil {
			return fmt.Errorf("failed to create share cpuset partition: %w", err)
		}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, SharePartition(), cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Run the Nomad client as root or grant CAP_SYS_ADMIN so /sys/fs/cgroup/cpuset/nomad is writable.
2. Verify /sys/fs/cgroup/cpuset exists and is mounted rw (mount | grep cgroup); remount with the cpuset controller enabled.
3. Inside containers, mount the host cgroup filesystem read-write or run with --privileged / cgroupns=host.
4. Check that the configured NomadCgroupParent directory exists and the kernel is actually using cgroup v1 for this controller.

Example fix

// before: run agent as unprivileged user
$ nomad agent -client
// error: failed to set clone_children on nomad cpuset cgroup: open /sys/fs/cgroup/cpuset/nomad/cgroup.clone_children: permission denied
// after
$ sudo nomad agent -client   # or grant CAP_SYS_ADMIN to the container

// before starting the Nomad client (cgroup v1)
if [ ! -w /sys/fs/cgroup/cpuset ]; then echo "cgroupfs cpuset not writable; run as root"; fi
mount | grep -E 'cgroup.*(cpuset)' || echo "cpuset controller not mounted"

err := cgroupslib.Init(logger, cores)
if err != nil && strings.Contains(err.Error(), "clone_children on nomad cpuset") {
    return fmt.Errorf("cgroup setup failed: run the agent as root or fix cgroup mounts: %w", err)
}

When it happens

Trigger: os.WriteFile on <cgroupRoot>/cpuset/<NomadCgroupParent>/cgroup.clone_children fails during Init when GetMode()==CG1, e.g. permission denied or the cpuset controller directory does not exist.

Common situations: Running the Nomad agent as a non-root user without the necessary capabilities; containerized Nomad agent without /sys/fs/cgroup mounted rw; kernel booted with cgroup_disable=cpuset or no cpuset controller mounted; cgroup v2 system where paths under /sys/fs/cgroup/cpuset do not exist.

Related errors

• failed to write cores to nomad cpuset cgroup: %w
• failed to create share cpuset partition: %w
• failed to set cpuset.mems on share cpuset partition: %w
• failed to create reserve cpuset partition: %w
• failed to set cpuset.mems on reserve cpuset partition: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d0cd67224f22f77c. Report an issue: GitHub.