{"record":{"id":"e1e9cb563a4c327c","repo":"hashicorp/nomad","slug":"failed-to-create-nomad-cgroup-w","errorCode":null,"errorMessage":"failed to create nomad cgroup: %w","messagePattern":"failed to create nomad cgroup: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/lib/cgroupslib/init.go","lineNumber":141,"sourceCode":"\n\t\tlog.Debug(\"nomad cpuset partitions initialized\", \"cores\", cores)\n\n\tcase CG2:\n\t\t// the cgroup controllers we need to activate at the root and on the nomad slice\n\t\tconst activation = \"+cpuset +cpu +io +memory +pids\"\n\n\t\t// the name of the cgroup subtree interface file\n\t\tconst subtreeFile = \"cgroup.subtree_control\"\n\n\t\t//\n\t\t// configuring root cgroup (/sys/fs/cgroup)\n\t\t//\n\t\t// clients with delegated cgroups typically won't be able to write to\n\t\t// the subtree file, but that's ok so long as the required controllers\n\t\t// are activated\n\t\tif !functionalCgroups2(subtreeFile) {\n\t\t\tif err := writeCG(activation, subtreeFile); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create nomad cgroup: %w\", err)\n\t\t\t}\n\t\t}\n\n\t\t//\n\t\t// configuring nomad.slice\n\t\t//\n\n\t\tif err := mkCG(NomadCgroupParent); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to create nomad cgroup: %w\", err)\n\t\t}\n\n\t\tif err := writeCG(activation, NomadCgroupParent, subtreeFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to set subtree control on nomad cgroup: %w\", err)\n\t\t}\n\n\t\tif err := writeCG(cores, NomadCgroupParent, cpusetFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write root partition cpuset: %w\", err)\n\t\t}","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/lib/cgroupslib/init.go#L123-L159","documentation":"During cgroupslib.Init in CG2 (cgroup v2 / unified hierarchy) mode, writing the controller activation string \"+cpuset +cpu +io +memory +pids\" to the ROOT cgroup.subtree_control file failed — but only when functionalCgroups2(subtreeFile) reported the controllers are not already functional. On systems with delegated cgroups (e.g. systemd user slices) the root write is expected to fail, so Nomad only attempts it when needed; when it fails and controllers are missing, nomad.slice cannot manage those controllers. Common wrapped errors: EACCES, EBUSY (processes running in root cgroup while enabling controllers), or ENOENT.","triggerScenarios":"functionalCgroups2(subtreeFile) returns false AND os.WriteFile on /sys/fs/cgroup/cgroup.subtree_control fails during Init CG2.","commonSituations":"Root cgroup has processes directly attached (cgroup v2 no-internal-process constraint → EBUSY); agent not running as root; container without writable /sys/fs/cgroup; kernel missing one of the controllers (+io on some kernels).","solutions":["Move all processes out of the root cgroup into child groups (usually done by booting via systemd) so subtree_control can be enabled — EBUSY is the classic blocker.","Run Nomad as root with CAP_SYS_ADMIN.","Ensure /sys/fs/cgroup is mounted rw in the container.","Verify each controller (cpuset, cpu, io, memory, pids) is available in /sys/fs/cgroup/cgroup.controllers; if io is absent, this indicates a kernel/config limitation.","Pre-provision nomad.slice via systemd so controllers are already delegated and functionalCgroups2 skips the root write."],"exampleFix":"// before: processes running directly in root cgroup\n$ cat /sys/fs/cgroup/cgroup.procs  # shows PIDs -> EBUSY on write\n// after: boot with systemd so all processes live in system.slice\n$ systemctl daemon-reload && sudo nomad agent -client","handlingStrategy":"validation","validationCode":"// pre-check for cgroup v2 delegation\nctls=$(cat /sys/fs/cgroup/cgroup.controllers)\nfor c in cpuset cpu io memory pids; do\n  case \" $ctls \" in *\" $c \"*) ;; *) echo \"controller $c unavailable at root\";; esac\nen=$(cat /sys/fs/cgroup/cgroup.subtree_control)\ncase \"$en\" in *\"+$c\"*) ;; *) echo \"$c not enabled in root subtree_control\";; esac\ndone\n# no-internal-process check: root cgroup must be empty of processes\n[ -z \"$(cat /sys/fs/cgroup/cgroup.procs)\" ] || echo \"processes in root cgroup: subtree_control write will fail with EBUSY\"","typeGuard":null,"tryCatchPattern":"if err := cgroupslib.Init(logger, cores); err != nil {\n    if strings.Contains(err.Error(), \"failed to create nomad cgroup\") && errors.Is(err, syscall.EBUSY) {\n        return fmt.Errorf(\"move processes out of the root cgroup (boot via systemd) and retry: %w\", err)\n    }\n    return err\n}","preventionTips":["Boot hosts with systemd so no processes sit directly in the root cgroup.","Pre-delegate controllers to nomad.slice via a systemd slice unit.","Run the agent as root with writable /sys/fs/cgroup.","Verify all required controllers appear in cgroup.controllers (io may be absent on some kernels)."],"tags":["linux","cgroups","cgroup-v2","systemd","subtree-control"],"backgroundTag":"cgroup-subtree-control-ebusy","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}