hashicorp/nomad · critical

failed to create nomad cgroup: %w

Error message

failed to create nomad cgroup: %w

What it means

During cgroupslib.Init in CG2 (cgroup v2 / unified hierarchy) mode, writing the controller activation string "+cpuset +cpu +io +memory +pids" to the ROOT cgroup.subtree_control file failed — but only when functionalCgroups2(subtreeFile) reported the controllers are not already functional. On systems with delegated cgroups (e.g. systemd user slices) the root write is expected to fail, so Nomad only attempts it when needed; when it fails and controllers are missing, nomad.slice cannot manage those controllers. Common wrapped errors: EACCES, EBUSY (processes running in root cgroup while enabling controllers), or ENOENT.

Source

Thrown at client/lib/cgroupslib/init.go:141

		log.Debug("nomad cpuset partitions initialized", "cores", cores)

	case CG2:
		// the cgroup controllers we need to activate at the root and on the nomad slice
		const activation = "+cpuset +cpu +io +memory +pids"

		// the name of the cgroup subtree interface file
		const subtreeFile = "cgroup.subtree_control"

		//
		// configuring root cgroup (/sys/fs/cgroup)
		//
		// clients with delegated cgroups typically won't be able to write to
		// the subtree file, but that's ok so long as the required controllers
		// are activated
		if !functionalCgroups2(subtreeFile) {
			if err := writeCG(activation, subtreeFile); err != nil {
				return fmt.Errorf("failed to create nomad cgroup: %w", err)
			}
		}

		//
		// configuring nomad.slice
		//

		if err := mkCG(NomadCgroupParent); err != nil {
			return fmt.Errorf("failed to create nomad cgroup: %w", err)
		}

		if err := writeCG(activation, NomadCgroupParent, subtreeFile); err != nil {
			return fmt.Errorf("failed to set subtree control on nomad cgroup: %w", err)
		}

		if err := writeCG(cores, NomadCgroupParent, cpusetFile); err != nil {
			return fmt.Errorf("failed to write root partition cpuset: %w", err)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Move all processes out of the root cgroup into child groups (usually done by booting via systemd) so subtree_control can be enabled — EBUSY is the classic blocker.
2. Run Nomad as root with CAP_SYS_ADMIN.
3. Ensure /sys/fs/cgroup is mounted rw in the container.
4. Verify each controller (cpuset, cpu, io, memory, pids) is available in /sys/fs/cgroup/cgroup.controllers; if io is absent, this indicates a kernel/config limitation.
5. Pre-provision nomad.slice via systemd so controllers are already delegated and functionalCgroups2 skips the root write.

Example fix

// before: processes running directly in root cgroup
$ cat /sys/fs/cgroup/cgroup.procs  # shows PIDs -> EBUSY on write
// after: boot with systemd so all processes live in system.slice
$ systemctl daemon-reload && sudo nomad agent -client

// pre-check for cgroup v2 delegation
ctls=$(cat /sys/fs/cgroup/cgroup.controllers)
for c in cpuset cpu io memory pids; do
  case " $ctls " in *" $c "*) ;; *) echo "controller $c unavailable at root";; esac
en=$(cat /sys/fs/cgroup/cgroup.subtree_control)
case "$en" in *"+$c"*) ;; *) echo "$c not enabled in root subtree_control";; esac
done
# no-internal-process check: root cgroup must be empty of processes
[ -z "$(cat /sys/fs/cgroup/cgroup.procs)" ] || echo "processes in root cgroup: subtree_control write will fail with EBUSY"

if err := cgroupslib.Init(logger, cores); err != nil {
    if strings.Contains(err.Error(), "failed to create nomad cgroup") && errors.Is(err, syscall.EBUSY) {
        return fmt.Errorf("move processes out of the root cgroup (boot via systemd) and retry: %w", err)
    }
    return err
}

When it happens

Trigger: functionalCgroups2(subtreeFile) returns false AND os.WriteFile on /sys/fs/cgroup/cgroup.subtree_control fails during Init CG2.

Common situations: Root cgroup has processes directly attached (cgroup v2 no-internal-process constraint → EBUSY); agent not running as root; container without writable /sys/fs/cgroup; kernel missing one of the controllers (+io on some kernels).

Related errors

• failed to set subtree control on cpuset share partition: %w
• failed to set subtree control on cpuset reserve partition: %
• ErrCgroupMustBeSet
• failed to create nomad cgroup %s: %w
• failed to set cpuset.mems on nomad cpuset cgroup: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e1e9cb563a4c327c. Report an issue: GitHub.