hashicorp/nomad · error

unable to get orphaned task PIDs: %v

Error message

unable to get orphaned task PIDs: %v

What it means

cleanOldProcessesInCGroup reads cgroup.procs via cgroups.GetAllPids to find orphaned processes left in the task's cgroup from a previous launch. If reading the PIDs fails and the error is NOT os.IsNotExist (i.e. the cgroup dir exists but its pid files are unreadable/broken), it returns this error. Missing cgroup is tolerated; unreadable cgroup is not.

Source

Thrown at drivers/shared/executor/executor_linux_cgo.go:148

	go le.catchSignals()

	le.processStats = procstats.New(compute, le)
	return le
}

func (l *LibcontainerExecutor) ListProcesses() set.Collection[int] {
	return procstats.List(l.command)
}

// cleanOldProcessesInCGroup kills processes that might ended up orphans when
// the executor was unexpectedly killed and nomad can't reconnect to them.
func (l *LibcontainerExecutor) cleanOldProcessesInCGroup(nomadRelativePath string) error {
	l.logger.Debug("looking for old processes", "path", nomadRelativePath)

	root := cgroupslib.GetDefaultRoot()
	orphanedPIDs, err := cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))
	if err != nil && !os.IsNotExist(err) {
		return fmt.Errorf("unable to get orphaned task PIDs: %v", err)
	}

	for _, pid := range orphanedPIDs {
		l.logger.Info("killing orphaned process", "pid", pid)

		// Avoid bringing down the whole node by mistake, very unlikely case,
		// but it's better to be sure.
		if pid == 1 {
			continue
		}

		if err := syscall.Kill(pid, syscall.SIGKILL); err != nil {
			return fmt.Errorf("unable to send signal to process %d: %v", pid, err)
		}
	}

	if len(orphanedPIDs) == 0 {
		return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the cgroup path (cat <root>/<nomadRelativePath>/cgroup.procs) as the Nomad user to reproduce the read failure
2. Fix mount/ownership permissions on the cgroup hierarchy used by Nomad (v1 vs v2, systemd driver)
3. Verify nomad's cgroups_v2 configuration and that GetDefaultRoot matches the actually mounted hierarchy
4. Restart nomad to recreate the cgroup structure if the path is stale/corrupt

pidFile := filepath.Join(cgroupslib.GetDefaultRoot(), nomadRelativePath, "cgroup.procs")
if _, err := os.Stat(pidFile); err == nil {
    if f, err := os.Open(pidFile); err != nil {
        // cgroup exists but unreadable: fix perms before Launch
        _ = f
    }
}

if err := executor.Launch(cmd); err != nil && strings.Contains(err.Error(), "unable to get orphaned task PIDs") {
    // inspect cgroup perms / mount, repair or recreate cgroup, then retry once
    log.Printf("cgroup pids unreadable: %v", err)
}

When it happens

Trigger: cgroups.GetAllPids(filepath.Join(cgroupslib.GetDefaultRoot(), nomadRelativePath)) fails with an error other than file-not-exist: permission denied on cgroup.procs, malformed cgroup filesystem, or I/O error reading the pid file.

Common situations: Cgroup mount permissions tightened (e.g. cgroup v2 paths restricted to root); the cgroup path exists but is corrupt or partially deleted; Nomad's cgroups root path misconfigured (nomad data dir vs /sys/fs/cgroup mismatch); kernel/cgroup driver changed between Nomad restarts.

Related errors

• ErrCgroupMustBeSet
• unable to send signal to process %d: %v
• orphaned processes %v have not been removed from cgroups pid
• failed to set cpuset: %w
• failed to create nomad cgroup %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0bacab154a6c3c6a. Report an issue: GitHub.