hashicorp/nomad · error

failed to set subtree control on nomad cgroup

Error message

failed to set subtree control on nomad cgroup: %w

What it means

During cgroups v2 client initialization, Nomad creates its top-level nomad.slice cgroup and writes the controller activation list (+cpuset +memory, etc.) to its cgroup.subtree_control file. This error wraps any failure writing that file, meaning Nomad cannot enable controllers for child cgroups and task isolation will not work.

Solutions

  1. Run the Nomad agent as root (or with CAP_SYS_ADMIN / appropriate cgroup delegation) so it can write to /sys/fs/cgroup/nomad.slice/cgroup.subtree_control.
  2. Check that the needed controllers are listed in /sys/fs/cgroup/cgroup.controllers and not already enabled in child cgroups (EBUSY); disable conflicting children or reboot.
  3. Verify /sys/fs/cgroup is mounted rw (mount | grep cgroup); remount without 'ro' if needed.
  4. If using systemd cgroup delegation, delegate the subtree to the Nomad service (Delegate=yes) and configure Nomad to use the delegated path.

Example fix

// before
sudo nomad agent -config /etc/nomad.d
// after (systemd unit with cgroup delegation)
[Service]
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
Delegate=yes
CapabilityBoundingSet=CAP_SYS_ADMIN
Defensive patterns

Strategy: validation

Validate before calling

func canWriteSubtreeControl() error {
  b, err := os.ReadFile("/sys/fs/cgroup/cgroup.controllers")
  if err != nil { return err }
  if len(b) == 0 { return errors.New("no controllers available at root cgroup") }
  return syscall.Access("/sys/fs/cgroup/nomad.slice", syscall.W_OK)
}

Type guard

func cgroupWritable(path string) bool {
  return syscall.Access(filepath.Join(path, "cgroup.subtree_control"), syscall.W_OK) == nil
}

Try / catch

if err := cgroupslib.Init(logger, cores); err != nil {
  var perr *fs.PathError
  if errors.As(err, &perr) && errors.Is(perr.Err, syscall.EACCES) {
    logger.Error("run nomad as root or delegate cgroups (systemd Delegate=yes)", "path", perr.Path)
  }
  return err
}

Prevention

When it happens

Trigger: Init() is called on a cgroups v2 host and os.WriteFile on /sys/fs/cgroup/nomad.slice/cgroup.subtree_control fails — typically EACCES (process lacks write access to the root cgroup), EBUSY (a controller already enabled in a child blocks activation), or the nomad.slice directory was created but is read-only.

Common situations: Nomad agent run as non-root or in a container without cgroup write privileges; systemd owns the root cgroup and delegates only a subtree to Nomad (delegated cgroups); cgroup filesystem mounted read-only; controllers not available in /sys/fs/cgroup/cgroup.controllers so activation fails.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ebc14352a0dd8359. Report an issue: GitHub.

Appendix: source

Thrown at client/lib/cgroupslib/init.go:154

		// clients with delegated cgroups typically won't be able to write to
		// the subtree file, but that's ok so long as the required controllers
		// are activated
		if !functionalCgroups2(subtreeFile) {
			if err := writeCG(activation, subtreeFile); err != nil {
				return fmt.Errorf("failed to create nomad cgroup: %w", err)
			}
		}

		//
		// configuring nomad.slice
		//

		if err := mkCG(NomadCgroupParent); err != nil {
			return fmt.Errorf("failed to create nomad cgroup: %w", err)
		}

		if err := writeCG(activation, NomadCgroupParent, subtreeFile); err != nil {
			return fmt.Errorf("failed to set subtree control on nomad cgroup: %w", err)
		}

		if err := writeCG(cores, NomadCgroupParent, cpusetFile); err != nil {
			return fmt.Errorf("failed to write root partition cpuset: %w", err)
		}

		log.Debug("top level partition root nomad.slice cgroup initialized")

		//
		// configuring nomad.slice/share (member)
		//

		if err := mkCG(NomadCgroupParent, SharePartition()); err != nil {
			return fmt.Errorf("failed to create share cgroup: %w", err)
		}

		if err := writeCG(activation, NomadCgroupParent, SharePartition(), subtreeFile); err != nil {
			return fmt.Errorf("failed to set subtree control on cpuset share partition: %w", err)

View on GitHub (pinned to 482b49bf1a)