hashicorp/nomad · critical

failed to set clone_children on nomad cpuset cgroup: %w

Error message

failed to set clone_children on nomad cpuset cgroup: %w

What it means

cgroupslib.Init fails when writeCG cannot write the clone_children flag (cgroup.clone_children = 1) to the Nomad cpuset cgroup. Nomad sets CLONE_CHILDREN so child cgroups inherit cpus/mems; failure here means the Nomad cgroup hierarchy cannot be configured and client startup aborts.

Source

Thrown at client/lib/cgroupslib/init.go:81

		}

		//
		// configure cpuset partitioning
		//
		// the tree is lopsided - tasks making use of reserved cpu cores get
		// their own cgroup with a static cpuset.cpus value. other tasks are
		// placed in the single share cgroup and share its dynamic cpuset.cpus
		// value
		//
		// e.g.,
		//  root/cpuset/nomad/
		//    share/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//    reserve/
		//      abc123.task/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//      def456.task/{cgroup.procs, cpuset.cpus, cpuset.mems}

		if err := writeCG(noClone, "cpuset", NomadCgroupParent, cloneFile); err != nil {
			return fmt.Errorf("failed to set clone_children on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(memsSet, "cpuset", NomadCgroupParent, memsFile); err != nil {
			return fmt.Errorf("failed to set cpuset.mems on nomad cpuset cgroup: %w", err)
		}

		if err := writeCG(cores, "cpuset", NomadCgroupParent, cpusetFile); err != nil {
			return fmt.Errorf("failed to write cores to nomad cpuset cgroup: %w", err)
		}

		//
		// share partition
		//

		if err := mkCG("cpuset", NomadCgroupParent, SharePartition()); err != nil {
			return fmt.Errorf("failed to create share cpuset partition: %w", err)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped error (%w) for ENOENT vs EACCES vs EROFS and address accordingly
2. Remount cgroupfs read-write or ensure the client runs with sufficient privileges
3. Recreate the /sys/fs/cgroup/cpuset/nomad directory if missing
4. Check SELinux/AppAudit policies denying writes to cgroup files (ausearch/sealert)
5. Avoid external managers (systemd delegation) claiming Nomad's cgroup parent

Example fix

// before
$ mount | grep cgroup  # cgroup mounted ro
// after
$ sudo mount -o remount,rw /sys/fs/cgroup
$ echo 1 > /sys/fs/cgroup/cpuset/nomad/cgroup.clone_children

f := "/sys/fs/cgroup/cpuset/nomad/cgroup.clone_children"
if err := os.WriteFile(f, []byte("1"), 0644); err != nil {
    return fmt.Errorf("cannot write clone_children at %s: %w", f, err)
}

func cloneChildrenWritable() bool {
    fi, err := os.Stat("/sys/fs/cgroup/cpuset/nomad/cgroup.clone_children")
    return err == nil && fi.Mode().Perm()&0o200 != 0
}

if err := cgroupslib.Init(cfg); err != nil {
    if strings.Contains(err.Error(), "failed to set clone_children") {
        log.Error("cgroupfs read-only or permissions denied; remount rw or run as root")
        return err
    }
    return err
}

When it happens

Trigger: Init (cgroup v1 path, newCG1) invokes writeCG(noClone, "cpuset", NomadCgroupParent, cloneFile) and the write fails: /sys/fs/cgroup/cpuset/nomad/cgroup.clone_children missing or read-only, or insufficient permissions to write.

Common situations: The nomad cgroup directory was created but cgroupfs is mounted read-only; client in a restricted container; another process (systemd) changed cgroup ownership/permissions; kernel lacking clone_children support (very old kernels); SELinux/AppArmor denying cgroup writes.

Related errors

• failed to create nomad cgroup %s: %w
• failed to detect memset: %w
• failed to set cpuset.mems on nomad cpuset cgroup: %w
• failed to write cores to nomad cpuset cgroup: %w
• failed to create share cpuset partition: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d32d4daa1b119063. Report an issue: GitHub.