hashicorp/nomad · critical

failed to detect memset: %w

Error message

failed to detect memset: %w

What it means

cgroupslib.Init fails when detectMemsCG1 cannot read the cpuset.mems value for the Nomad cgroup parent. Nomad needs the allowed memory-node mask to configure cpuset partitioning for tasks; without it, client initialization aborts.

Source

Thrown at client/lib/cgroupslib/init.go:60

		controllers := []string{"freezer", "memory", "cpu", "cpuset"}
		for _, ctrl := range controllers {
			p := filepath.Join(root, ctrl, NomadCgroupParent)
			if err := os.MkdirAll(p, 0755); err != nil {
				return fmt.Errorf("failed to create nomad cgroup %s: %w", ctrl, err)
			}
		}

		// determine the memset that will be set on the cgroup for each task
		//
		// nominally this will be all available but we have to read the root
		// cgroup to actually know what those are
		//
		// additionally if the nomad cgroup parent already exists, we must
		// use that memset instead, because it could have been setup out of
		// band from nomad itself
		var memsSet string
		if mems, err := detectMemsCG1(); err != nil {
			return fmt.Errorf("failed to detect memset: %w", err)
		} else {
			memsSet = mems
		}

		//
		// configure cpuset partitioning
		//
		// the tree is lopsided - tasks making use of reserved cpu cores get
		// their own cgroup with a static cpuset.cpus value. other tasks are
		// placed in the single share cgroup and share its dynamic cpuset.cpus
		// value
		//
		// e.g.,
		//  root/cpuset/nomad/
		//    share/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//    reserve/
		//      abc123.task/{cgroup.procs, cpuset.cpus, cpuset.mems}
		//      def456.task/{cgroup.procs, cpuset.cpus, cpuset.mems}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the cpuset controller is mounted and /sys/fs/cgroup/cpuset/cpuset.mems is readable
2. Check the underlying error (%w) — usually a stat/read permission or missing-file problem
3. Run the client as root or fix cgroup file permissions
4. If in a container, mount /sys/fs/cgroup read-write or use --cgroupns=host
5. Confirm kernel supports/has cpuset enabled (CONFIG_CPUSETS)

Example fix

// before
$ cat /sys/fs/cgroup/cpuset/cpuset.mems  # No such file or directory
// after
$ sudo mount -t cgroup -o cpuset none /sys/fs/cgroup/cpuset
$ cat /sys/fs/cgroup/cpuset/cpuset.mems  # 0

b, err := os.ReadFile("/sys/fs/cgroup/cpuset/cpuset.mems")
if err != nil {
    return fmt.Errorf("cpuset.mems unreadable, cpuset controller missing or no permission: %w", err)
}
if strings.TrimSpace(string(b)) == "" {
    return fmt.Errorf("cpuset.mems empty")
}

func memsReadable() bool {
    _, err := os.ReadFile("/sys/fs/cgroup/cpuset/cpuset.mems")
    return err == nil
}

if err := cgroupslib.Init(cfg); err != nil {
    if strings.Contains(err.Error(), "failed to detect memset") {
        log.Error("cpuset.mems unavailable; ensure cpuset controller mounted and readable")
        // abort or fall back to non-cpuset mode
    }
    return err
}

When it happens

Trigger: Init (cgroup v1 path) calls detectMemsCG1 which fails to read/derive cpuset.mems — e.g. cpuset controller not mounted, /sys/fs/cgroup/cpuset/nomad/cpuset.mems unreadable, or the root cpuset.mems file missing/unreadable.

Common situations: cpuset cgroup controller not mounted on the host; Nomad client running in a container without access to the host cgroup filesystem; unusual NUMA configurations; kernel/boot options disabling cpuset; permissions on cgroup files after systemd changes.

Related errors

• failed to set clone_children on nomad cpuset cgroup: %w
• failed to write cores to nomad cpuset cgroup: %w
• failed to set cpuset.mems on share cpuset partition: %w
• failed to set cpuset.mems on reserve cpuset partition: %w
• failed to create nomad cgroup %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/41ffeff3d93b7bbc. Report an issue: GitHub.