{"record":{"id":"c14cb127107ffa2c","repo":"hashicorp/nomad","slug":"failed-to-set-cpuset-mems-on-reserve-cpuset-partit","errorCode":null,"errorMessage":"failed to set cpuset.mems on reserve cpuset partition: %w","messagePattern":"failed to set cpuset\\.mems on reserve cpuset partition: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/lib/cgroupslib/init.go","lineNumber":121,"sourceCode":"\n\t\tif err := writeCG(memsSet, \"cpuset\", NomadCgroupParent, SharePartition(), memsFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to set cpuset.mems on share cpuset partition: %w\", err)\n\t\t}\n\n\t\t//\n\t\t// reserve partition\n\t\t//\n\n\t\tif err := mkCG(\"cpuset\", NomadCgroupParent, ReservePartition()); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to create reserve cpuset partition: %w\", err)\n\t\t}\n\n\t\tif err := writeCG(noClone, \"cpuset\", NomadCgroupParent, ReservePartition(), cloneFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to set clone_children on nomad cpuset cgroup: %w\", err)\n\t\t}\n\n\t\tif err := writeCG(memsSet, \"cpuset\", NomadCgroupParent, ReservePartition(), memsFile); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to set cpuset.mems on reserve cpuset partition: %w\", err)\n\t\t}\n\n\t\tlog.Debug(\"nomad cpuset partitions initialized\", \"cores\", cores)\n\n\tcase CG2:\n\t\t// the cgroup controllers we need to activate at the root and on the nomad slice\n\t\tconst activation = \"+cpuset +cpu +io +memory +pids\"\n\n\t\t// the name of the cgroup subtree interface file\n\t\tconst subtreeFile = \"cgroup.subtree_control\"\n\n\t\t//\n\t\t// configuring root cgroup (/sys/fs/cgroup)\n\t\t//\n\t\t// clients with delegated cgroups typically won't be able to write to\n\t\t// the subtree file, but that's ok so long as the required controllers\n\t\t// are activated\n\t\tif !functionalCgroups2(subtreeFile) {","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/lib/cgroupslib/init.go#L103-L139","documentation":"During cgroupslib.Init in CG1 mode, writing the mems value to /sys/fs/cgroup/cpuset/<nomad-parent>/reserve/cpuset.mems failed. The reserve partition needs a valid memory-node mask before reserved-core task cgroups are created beneath it. Failures mirror the share partition case: permission errors, read-only mounts, or an invalid/empty mems value.","triggerScenarios":"os.WriteFile on the reserve partition's cpuset.mems with the detected memsSet fails during Init CG1 — the final cpuset write before Init logs success.","commonSituations":"Empty root cpuset.mems producing an invalid value; non-root agent; container with read-only cgroup mount; NUMA topology changes after an out-of-band parent mems value was cached.","solutions":["Check root cpuset.mems holds a valid mask and the memsSet value matches allowed nodes.","Run Nomad with privileges to write cgroupfs.","Ensure cgroupfs is mounted rw.","Delete a stale/incompletely configured nomad cpuset parent and let Init rebuild it."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"mems=$(cat /sys/fs/cgroup/cpuset/cpuset.mems 2>/dev/null)\n[ -n \"$mems\" ] || echo \"root cpuset.mems empty; fix before starting Nomad\"\ncat /sys/fs/cgroup/cpuset/nomad/reserve/cpuset.mems 2>/dev/null || echo \"reserve cpuset.mems missing\"","typeGuard":null,"tryCatchPattern":"if err := cgroupslib.Init(logger, cores); err != nil {\n    if strings.Contains(err.Error(), \"cpuset.mems on reserve\") {\n        return fmt.Errorf(\"failed to pin mems on reserve partition; verify mems mask and privileges: %w\", err)\n    }\n    return err\n}","preventionTips":["Validate the root cpuset.mems mask before agent startup.","Run with sufficient cgroupfs write privileges.","Re-init cleanly if the nomad parent was configured out of band with stale mems."],"tags":["linux","cgroups","cgroup-v1","cpuset","numa"],"backgroundTag":"cgroup-cpuset-invalid-mems","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}