hashicorp/nomad · critical

failed to write cores to nomad cpuset cgroup: %w

Error message

failed to write cores to nomad cpuset cgroup: %w

What it means

During cgroupslib.Init in CG1 mode, writing the detected mems value to /sys/fs/cgroup/cpuset/<nomad-parent>/cpuset.mems failed. Nomad pins the parent cgroup's cpuset.mems to the memory nodes read from the root cpuset (or the existing parent value). Failures are usually EACCES (insufficient privileges) or EINVAL (value not compatible with the parent's allowed mems).

Source

Thrown at client/lib/cgroupslib/init.go:89

		// value
		//
		// e.g.,
		//  root/cpuset/nomad/
		//    share/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//    reserve/
		//      abc123.task/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//      def456.task/{cgroup.procs, cpuset.cpus, cpuset.mems}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(cores, "cpuset", NomadCgroupParent, cpusetFile); err != nil {
			return fmt.Errorf("failed to write cores to nomad cpuset cgroup: %w", err)
		}

		//
		// share partition
		//

		if err := mkCG("cpuset", NomadCgroupParent, SharePartition()); err != nil {
			return fmt.Errorf("failed to create share cpuset partition: %w", err)
		}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, SharePartition(), cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, SharePartition(), memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on share cpuset partition: %w", err)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Run Nomad with root privileges / CAP_SYS_ADMIN so cgroupfs writes succeed.
2. Ensure /sys/fs/cgroup is mounted read-write in the container or host environment.
3. Verify the value being written is a valid mems mask (compare with cat /sys/fs/cgroup/cpuset/cpuset.mems).
4. If the nomad cgroup was pre-created by systemd or tooling, align its cpuset.mems with the root value or delete it so Nomad initializes it fresh.

mems=$(cat /sys/fs/cgroup/cpuset/cpuset.mems 2>/dev/null)
if [ -z "$mems" ]; then echo "root cpuset.mems empty; cpuset controller not ready"; fi
if [ ! -w /sys/fs/cgroup/cpuset/nomad/cpuset.mems ]; then echo "cannot write cpuset.mems; need root"; fi

if err := cgroupslib.Init(logger, cores); err != nil {
    if strings.Contains(err.Error(), "cpuset.mems on nomad cpuset") {
        logger.Error("failed to pin mems on nomad cgroup; check root cpuset.mems and privileges", "err", err)
    }
    return err
}

When it happens

Trigger: os.WriteFile on <cgroupRoot>/cpuset/<NomadCgroupParent>/cpuset.mems with the memsSet value fails during Init CG1, right after clone_children was set successfully.

Common situations: Non-root agent lacking write access to cgroupfs; read-only /sys/fs/cgroup mount in a container; an out-of-band parent cpuset.mems value conflicting with what Nomad tries to write; cpuset controller not fully initialized so writes to empty cpuset.mems are rejected.

Related errors

• failed to set cpuset.mems on share cpuset partition: %w
• failed to set cpuset.mems on reserve cpuset partition: %w
• failed to set cpuset.mems on nomad cpuset cgroup: %w
• failed to create share cpuset partition: %w
• failed to create reserve cpuset partition: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/72edd03b95b9c494. Report an issue: GitHub.