hashicorp/nomad · error

failed to set subtree control on cpuset reserve partition: %

Error message

failed to set subtree control on cpuset reserve partition: %w

What it means

After creating nomad.slice/reserve, Init() writes the controller activation list to its cgroup.subtree_control so per-task cgroups beneath reserve can use the controllers. Failure leaves the reserve partition unable to host task cgroups.

Source

Thrown at client/lib/cgroupslib/init.go:186

			return fmt.Errorf("failed to create share cgroup: %w", err)
		}

		if err := writeCG(activation, NomadCgroupParent, SharePartition(), subtreeFile); err != nil {
			return fmt.Errorf("failed to set subtree control on cpuset share partition: %w", err)
		}

		log.Debug("partition member nomad.slice/share cgroup initialized")

		//
		// configuring nomad.slice/reserve (member)
		//

		if err := mkCG(NomadCgroupParent, ReservePartition()); err != nil {
			return fmt.Errorf("failed to create share cgroup: %w", err)
		}

		if err := writeCG(activation, NomadCgroupParent, ReservePartition(), subtreeFile); err != nil {
			return fmt.Errorf("failed to set subtree control on cpuset reserve partition: %w", err)
		}

		log.Debug("partition member nomad.slice/reserve cgroup initialized")
	}

	return nil
}

// detectMemsCG1 will determine the cpuset.mems value to use for
// Nomad managed cgroups.
//
// Copy the value from the root cgroup cpuset.mems file, unless the nomad
// parent cgroup exists with a value set, in which case use the cpuset.mems
// value from there.
func detectMemsCG1() (string, error) {
	// read root cgroup mems file
	memsRootPath := filepath.Join(root, "cpuset", memsFile)
	b, err := os.ReadFile(memsRootPath)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure nomad.slice/reserve/cgroup.procs is empty before starting Nomad (stop tasks/agent first).
2. Run as root or with systemd cgroup delegation so subtree_control is writable.
3. Verify required controllers appear in /sys/fs/cgroup/cgroup.controllers; upgrade kernel or boot config if absent.
4. Check the cgroup mount is not read-only.

Example fix

# before: EBUSY due to live processes
$ cat /sys/fs/cgroup/nomad.slice/reserve/cgroup.procs
4321
// after: stop nomad, drain tasks, then restart
$ sudo systemctl stop nomad; kill 4321; sudo systemctl start nomad

func reserveIsEmpty(reserveDir string) (bool, error) {
  b, err := os.ReadFile(filepath.Join(reserveDir, "cgroup.procs"))
  if err != nil { return false, err }
  return len(strings.TrimSpace(string(b))) == 0, nil
}

func controllersAvailable(needed []string) bool {
  b, _ := os.ReadFile("/sys/fs/cgroup/cgroup.controllers")
  have := strings.Fields(string(b))
  for _, n := range needed {
    if !slices.Contains(have, n) { return false }
  }
  return true
}

if err := cgroupslib.Init(logger, cores); err != nil {
  if strings.Contains(err.Error(), "subtree control on cpuset reserve") {
    // likely EBUSY from live task processes; drain then retry
    return retryAfterDrain(err)
  }
  return err
}

When it happens

Trigger: writeCG(activation, NomadCgroupParent, ReservePartition(), subtreeFile) fails — EACCES writing the file, EBUSY because processes already reside in nomad.slice/reserve, or the controllers to activate are unavailable on this kernel.

Common situations: Leftover reserve cgroup with running task processes from a prior Nomad run; agent without cgroup write access; kernel lacking a controller Nomad activates (e.g. cpu controller unavailable due to cgroup_no_v1 settings or kernel version).

Related errors

• failed to set subtree control on cpuset share partition: %w
• failed to create nomad cgroup: %w
• failed to set subtree control on nomad cgroup: %w
• failed to write root partition cpuset: %w
• failed to create share cgroup: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/cef1c10c8d9cbdc4. Report an issue: GitHub.