{"record":{"id":"d0cd67224f22f77c","repo":"hashicorp/nomad","slug":"failed-to-set-cpuset-mems-on-nomad-cpuset-cgroup","errorCode":null,"errorMessage":"failed to set cpuset.mems on nomad cpuset cgroup: %w","messagePattern":"failed to set cpuset\\.mems on nomad cpuset cgroup: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/lib/cgroupslib/init.go","lineNumber":85,"sourceCode":"\t\t//\n\t\t// the tree is lopsided - tasks making use of reserved cpu cores get\n\t\t// their own cgroup with a static cpuset.cpus value. other tasks are\n\t\t// placed in the single share cgroup and share its dynamic cpuset.cpus\n\t\t// value\n\t\t//\n\t\t// e.g.,\n\t\t//  root/cpuset/nomad/\n\t\t//    share/{cgroup.procs, cpuset.cpus, cpuset.mems}\n\t\t//    reserve/\n\t\t//      abc123.task/{cgroup.procs, cpuset.cpus, cpuset.mems}\n\t\t//      def456.task/{cgroup.procs, cpuset.cpus, cpuset.mems}\n\n\t\tif err := writeCG(noClone, \"cpuset\", NomadCgroupParent, cloneFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to set clone_children on nomad cpuset cgroup: %w\", err)\n\t\t}\n\n\t\tif err := writeCG(memsSet, \"cpuset\", NomadCgroupParent, memsFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to set cpuset.mems on nomad cpuset cgroup: %w\", err)\n\t\t}\n\n\t\tif err := writeCG(cores, \"cpuset\", NomadCgroupParent, cpusetFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write cores to nomad cpuset cgroup: %w\", err)\n\t\t}\n\n\t\t//\n\t\t// share partition\n\t\t//\n\n\t\tif err := mkCG(\"cpuset\", NomadCgroupParent, SharePartition()); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to create share cpuset partition: %w\", err)\n\t\t}\n\n\t\tif err := writeCG(noClone, \"cpuset\", NomadCgroupParent, SharePartition(), cloneFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to set clone_children on nomad cpuset cgroup: %w\", err)\n\t\t}\n","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/lib/cgroupslib/init.go#L67-L103","documentation":"During cgroupslib.Init in CG1 (cgroup v1) mode, writing the value \"0\" to /sys/fs/cgroup/cpuset/<nomad-parent>/cgroup.clone_children failed. Nomad sets clone_children=0 on its parent cpuset cgroup so that child cgroups do not automatically inherit cpus/mems and instead get explicit values. The write is done via os.WriteFile, so the wrapped error is typically EACCES (not root / no CAP_SYS_ADMIN) or ENOENT (cgroup controller not mounted).","triggerScenarios":"os.WriteFile on <cgroupRoot>/cpuset/<NomadCgroupParent>/cgroup.clone_children fails during Init when GetMode()==CG1, e.g. permission denied or the cpuset controller directory does not exist.","commonSituations":"Running the Nomad agent as a non-root user without the necessary capabilities; containerized Nomad agent without /sys/fs/cgroup mounted rw; kernel booted with cgroup_disable=cpuset or no cpuset controller mounted; cgroup v2 system where paths under /sys/fs/cgroup/cpuset do not exist.","solutions":["Run the Nomad client as root or grant CAP_SYS_ADMIN so /sys/fs/cgroup/cpuset/nomad is writable.","Verify /sys/fs/cgroup/cpuset exists and is mounted rw (mount | grep cgroup); remount with the cpuset controller enabled.","Inside containers, mount the host cgroup filesystem read-write or run with --privileged / cgroupns=host.","Check that the configured NomadCgroupParent directory exists and the kernel is actually using cgroup v1 for this controller."],"exampleFix":"// before: run agent as unprivileged user\n$ nomad agent -client\n// error: failed to set clone_children on nomad cpuset cgroup: open /sys/fs/cgroup/cpuset/nomad/cgroup.clone_children: permission denied\n// after\n$ sudo nomad agent -client   # or grant CAP_SYS_ADMIN to the container","handlingStrategy":"validation","validationCode":"// before starting the Nomad client (cgroup v1)\nif [ ! -w /sys/fs/cgroup/cpuset ]; then echo \"cgroupfs cpuset not writable; run as root\"; fi\nmount | grep -E 'cgroup.*(cpuset)' || echo \"cpuset controller not mounted\"","typeGuard":null,"tryCatchPattern":"err := cgroupslib.Init(logger, cores)\nif err != nil && strings.Contains(err.Error(), \"clone_children on nomad cpuset\") {\n    return fmt.Errorf(\"cgroup setup failed: run the agent as root or fix cgroup mounts: %w\", err)\n}","preventionTips":["Run the Nomad client as root or with CAP_SYS_ADMIN.","Verify cpuset is mounted in a cgroup v1 host before booting the agent.","In containers, mount /sys/fs/cgroup read-write and use cgroupns=host."],"tags":["linux","cgroups","cgroup-v1","cpuset","permissions"],"backgroundTag":"cgroup-write-permission-denied","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}