hashicorp/nomad · critical

failed to create nomad cgroup %s: %w

Error message

failed to create nomad cgroup %s: %w

What it means

cgroupslib.Init fails when it cannot create the /nomad cgroup directory (NomadCgroupParent) under each required controller (freezer, memory, cpu, cpuset) via os.MkdirAll. This means Nomad cannot set up its cgroup hierarchy for task resource isolation and client startup aborts. The wrapped error names which controller failed.

Source

Thrown at client/lib/cgroupslib/init.go:46

func Init(log hclog.Logger, cores string) error {
	log.Info("initializing nomad cgroups", "cores", cores)

	switch GetMode() {
	case CG1:

		// the value to disable inheriting values from parent cgroup
		const noClone = "0"

		// the name of the clone_children interface file
		const cloneFile = "cgroup.clone_children"

		// create the /nomad cgroup (or whatever the name is configured to be)
		// for each cgroup controller we are going to use
		controllers := []string{"freezer", "memory", "cpu", "cpuset"}
		for _, ctrl := range controllers {
			p := filepath.Join(root, ctrl, NomadCgroupParent)
			if err := os.MkdirAll(p, 0755); err != nil {
				return fmt.Errorf("failed to create nomad cgroup %s: %w", ctrl, err)
			}
		}

		// determine the memset that will be set on the cgroup for each task
		//
		// nominally this will be all available but we have to read the root
		// cgroup to actually know what those are
		//
		// additionally if the nomad cgroup parent already exists, we must
		// use that memset instead, because it could have been setup out of
		// band from nomad itself
		var memsSet string
		if mems, err := detectMemsCG1(); err != nil {
			return fmt.Errorf("failed to detect memset: %w", err)
		} else {
			memsSet = mems
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure the Nomad client runs as root (or with cgroup write privileges)
2. Check controllers are mounted: ls /sys/fs/cgroup (v1) and confirm freezer/memory/cpu/cpuset exist
3. Remove cgroup_disable=memory (and similar) from kernel cmdline and reboot
4. Remount cgroup filesystem read-write or fix the container runtime flags (--cgroupns=host, privileged)
5. Verify NomadCgroupParent doesn't collide with a file/improper path under the controller

Example fix

// before
$ nomad agent -client  # as non-root: failed to create nomad cgroup memory: mkdir ... permission denied
// after
$ sudo nomad agent -client
# or in Docker: docker run --privileged --cgroupns=host ...

for _, c := range []string{"freezer", "memory", "cpu", "cpuset"} {
    if _, err := os.Stat(filepath.Join("/sys/fs/cgroup", c)); err != nil {
        return fmt.Errorf("controller %s not mounted: %w", c, err)
    }
}
if err := syscall.Access("/sys/fs/cgroup", os.O_RDWR); err != nil {
    return fmt.Errorf("cgroupfs not writable (need root?): %w", err)
}

func canManageCgroups() bool {
    return os.Geteuid() == 0 && isMounted("/sys/fs/cgroup")
}

if err := cgroupslib.Init(cfg); err != nil {
    var perr *fs.PathError
    if errors.As(err, &perr) {
        log.Error("cgroup controller unavailable", "path", perr.Path)
        // fall back to cgroup-v2 mode or fail fast with clear operator guidance
    }
    return err
}

When it happens

Trigger: Running Init (cgroup v1 path via newCG1, or v2 via newCG2) where /sys/fs/cgroup/<controller>/nomad cannot be created: cgroup controller not mounted, read-only cgroup filesystem, rootless/insufficient privileges, or containers restricting cgroup writes.

Common situations: Nomad client running in a Docker container without cgroup namespace privileges; cgroup v1 controller not mounted (e.g. memory controller disabled at kernel boot with cgroup_disable=memory); /sys/fs/cgroup mounted read-only; running client as non-root on hosts requiring root; systemd delegating cgroups restrictively.

Related errors

• failed to set clone_children on nomad cpuset cgroup: %w
• failed to set cpuset.mems on nomad cpuset cgroup: %w
• ErrCgroupMustBeSet
• Unable to find nobody user: %w
• Couldn't change owner/group of %v to (uid: %v, gid: %v): %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/dab70f188b79db47. Report an issue: GitHub.