hashicorp/nomad · error

unable to write to custom cgroup: %v

Error message

unable to write to custom cgroup: %v

What it means

configureCG1 writes the executor process PID into each cgroup v1 controller path the command overrides via OverrideCgroupV1. If writing cgroup.procs fails for any custom path, the error is logged and returned wrapped. It means a user-specified custom cgroup could not accept the process.

Source

Thrown at drivers/shared/executor/executor_universal_linux.go:197

}

func (e *UniversalExecutor) configureCG1(cgroup string, command *ExecCommand) error {
	// some drivers like qemu entirely own resource management
	if command.Resources == nil || command.Resources.LinuxResources == nil {
		return nil
	}

	// if custom cgroups are set join those instead of configuring the /nomad
	// cgroups we are not going to use
	if len(e.command.OverrideCgroupV1) > 0 {
		pid := unix.Getpid()
		for controller, path := range e.command.OverrideCgroupV1 {
			absPath := cgroupslib.CustomPathCG1(controller, path)
			ed := cgroupslib.OpenPath(absPath)
			err := ed.Write("cgroup.procs", strconv.Itoa(pid))
			if err != nil {
				e.logger.Error("unable to write to custom cgroup", "error", err)
				return fmt.Errorf("unable to write to custom cgroup: %v", err)
			}
		}
		return nil
	}

	// write memory limits
	memHard, memReserved := memoryLimits(command.Resources.NomadResources.Memory)
	ed := cgroupslib.OpenFromFreezerCG1(cgroup, "memory")
	_ = ed.Write("memory.limit_in_bytes", strconv.FormatInt(memHard, 10))
	if memReserved > 0 {
		_ = ed.Write("memory.soft_limit_in_bytes", strconv.FormatInt(memReserved, 10))
	}

	// write memory swappiness
	swappiness := cgroupslib.MaybeDisableMemorySwappiness()
	if swappiness != nil {
		value := int64(*swappiness)
		_ = ed.Write("memory.swappiness", strconv.FormatInt(value, 10))

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify each OverrideCgroupV1 path exists on the node and the controller is mounted (cgroupslib.CustomPathCG1 layout)
2. Grant the Nomad client user write access (ownership/permissions) to the custom cgroup directories
3. Create the custom cgroup before the task starts (systemd slice/cgcreate) and fix the config typo
4. Remove the override if the controller is unavailable on this host

Example fix

// before
OverrideCgroupV1 = {"hugetlb": "nomad/custom"}  # hugetlb not mounted
// after
# cgcreate -g hugetlb:/nomad/custom  (or drop the override)
OverrideCgroupV1 = {"memory": "nomad/custom"}

for _, path := range overrideCgroups {
    abs := cgroupslib.CustomPathCG1(controller, path)
    if _, err := os.Stat(filepath.Join(abs, "cgroup.procs")); err != nil {
        return fmt.Errorf("custom cgroup %s unavailable", abs)
    }
}

if err != nil && strings.Contains(err.Error(), "unable to write to custom cgroup") {
    // path is in client logs; create/fix the cgroup then reschedule
    return fmt.Errorf("fix OverrideCgroupV1 config: %w", err)
}

When it happens

Trigger: Task/driver config sets OverrideCgroupV1 with controller->path pairs; configureResourceContainer -> configureCG1 then tries to echo the PID into <abs>/cgroup.procs and the write fails (path missing, permissions, controller not mounted).

Common situations: Operator typo in custom cgroup path; cgroup path created by an external system not yet present at task start; permissions/delegation missing for the Nomad user; controller (e.g. hugetlb) not mounted on the node.

Related errors

• failed to set cpuset.mems on nomad cpuset cgroup: %w
• failed to write cores to nomad cpuset cgroup: %w
• failed to create share cpuset partition: %w
• failed to set cpuset.mems on share cpuset partition: %w
• failed to create reserve cpuset partition: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/529fbffc33bdda12. Report an issue: GitHub.