{"record":{"id":"41ffeff3d93b7bbc","repo":"hashicorp/nomad","slug":"failed-to-detect-memset-w","errorCode":null,"errorMessage":"failed to detect memset: %w","messagePattern":"failed to detect memset: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/lib/cgroupslib/init.go","lineNumber":60,"sourceCode":"\t\tcontrollers := []string{\"freezer\", \"memory\", \"cpu\", \"cpuset\"}\n\t\tfor _, ctrl := range controllers {\n\t\t\tp := filepath.Join(root, ctrl, NomadCgroupParent)\n\t\t\tif err := os.MkdirAll(p, 0755); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create nomad cgroup %s: %w\", ctrl, err)\n\t\t\t}\n\t\t}\n\n\t\t// determine the memset that will be set on the cgroup for each task\n\t\t//\n\t\t// nominally this will be all available but we have to read the root\n\t\t// cgroup to actually know what those are\n\t\t//\n\t\t// additionally if the nomad cgroup parent already exists, we must\n\t\t// use that memset instead, because it could have been setup out of\n\t\t// band from nomad itself\n\t\tvar memsSet string\n\t\tif mems, err := detectMemsCG1(); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to detect memset: %w\", err)\n\t\t} else {\n\t\t\tmemsSet = mems\n\t\t}\n\n\t\t//\n\t\t// configure cpuset partitioning\n\t\t//\n\t\t// the tree is lopsided - tasks making use of reserved cpu cores get\n\t\t// their own cgroup with a static cpuset.cpus value. other tasks are\n\t\t// placed in the single share cgroup and share its dynamic cpuset.cpus\n\t\t// value\n\t\t//\n\t\t// e.g.,\n\t\t//  root/cpuset/nomad/\n\t\t//    share/{cgroup.procs, cpuset.cpus, cpuset.mems}\n\t\t//    reserve/\n\t\t//      abc123.task/{cgroup.procs, cpuset.cpus, cpuset.mems}\n\t\t//      def456.task/{cgroup.procs, cpuset.cpus, cpuset.mems}","sourceCodeStart":42,"sourceCodeEnd":78,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/lib/cgroupslib/init.go#L42-L78","documentation":"cgroupslib.Init fails when detectMemsCG1 cannot read the cpuset.mems value for the Nomad cgroup parent. Nomad needs the allowed memory-node mask to configure cpuset partitioning for tasks; without it, client initialization aborts.","triggerScenarios":"Init (cgroup v1 path) calls detectMemsCG1 which fails to read/derive cpuset.mems — e.g. cpuset controller not mounted, /sys/fs/cgroup/cpuset/nomad/cpuset.mems unreadable, or the root cpuset.mems file missing/unreadable.","commonSituations":"cpuset cgroup controller not mounted on the host; Nomad client running in a container without access to the host cgroup filesystem; unusual NUMA configurations; kernel/boot options disabling cpuset; permissions on cgroup files after systemd changes.","solutions":["Verify the cpuset controller is mounted and /sys/fs/cgroup/cpuset/cpuset.mems is readable","Check the underlying error (%w) — usually a stat/read permission or missing-file problem","Run the client as root or fix cgroup file permissions","If in a container, mount /sys/fs/cgroup read-write or use --cgroupns=host","Confirm kernel supports/has cpuset enabled (CONFIG_CPUSETS)"],"exampleFix":"// before\n$ cat /sys/fs/cgroup/cpuset/cpuset.mems  # No such file or directory\n// after\n$ sudo mount -t cgroup -o cpuset none /sys/fs/cgroup/cpuset\n$ cat /sys/fs/cgroup/cpuset/cpuset.mems  # 0","handlingStrategy":"validation","validationCode":"b, err := os.ReadFile(\"/sys/fs/cgroup/cpuset/cpuset.mems\")\nif err != nil {\n    return fmt.Errorf(\"cpuset.mems unreadable, cpuset controller missing or no permission: %w\", err)\n}\nif strings.TrimSpace(string(b)) == \"\" {\n    return fmt.Errorf(\"cpuset.mems empty\")\n}","typeGuard":"func memsReadable() bool {\n    _, err := os.ReadFile(\"/sys/fs/cgroup/cpuset/cpuset.mems\")\n    return err == nil\n}","tryCatchPattern":"if err := cgroupslib.Init(cfg); err != nil {\n    if strings.Contains(err.Error(), \"failed to detect memset\") {\n        log.Error(\"cpuset.mems unavailable; ensure cpuset controller mounted and readable\")\n        // abort or fall back to non-cpuset mode\n    }\n    return err\n}","preventionTips":["Confirm the cpuset cgroup controller is mounted at boot","Run the client with read access to /sys/fs/cgroup/cpuset files","In containers, expose host cgroupfs (read-write for Nomad paths)","Validate NUMA setup on hosts with multiple memory nodes"],"tags":["cgroups","cpuset","numa","client-startup"],"backgroundTag":"cgroup-setup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}