orphaned processes %v have not been removed from cgroups pid

Error message

orphaned processes %v have not been removed from cgroups pid file

What it means

After sending SIGKILL to orphaned PIDs, the function polls (up to ~10 iterations with backoff sleeps) waiting for the PIDs to disappear from the cgroup pid file. If they are still present after the retries, it returns this error, refusing to launch a new task into a cgroup that still holds live processes.

Source

Thrown at drivers/shared/executor/executor_linux_cgo.go:180

		}
	}

	if len(orphanedPIDs) == 0 {
		return nil
	}

	// Make sure the PID was removed from the cgroup file, otherwise
	// libcontainer will not be able to launch. Five retries every 100 ms should be
	// more than enough.
	for i := 100; i < 501; i += 100 {
		orphanedPIDs, _ = cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))
		if len(orphanedPIDs) > 0 {
			time.Sleep(time.Duration(i) * time.Millisecond)
			continue
		}
		return nil
	}
	return fmt.Errorf("orphaned processes %v have not been removed from cgroups pid file", orphanedPIDs)
}

// Launch creates a new container in libcontainer and starts a new process with it
func (l *LibcontainerExecutor) Launch(command *ExecCommand) (*ProcessState, error) {
	l.logger.Trace("preparing to launch command", "command", command.Cmd, "args", strings.Join(command.Args, " "))

	if command.Resources == nil {
		command.Resources = &drivers.Resources{
			NomadResources: &structs.AllocatedTaskResources{},
		}
	}

	l.command = command

	// A container groups processes under the same isolation enforcement
	containerCfg, err := l.newLibcontainerConfig(command)
	if err != nil {
		return nil, fmt.Errorf("failed to configure container(%s): %v", l.id, err)

View on GitHub (pinned to 482b49bf1a)

Solutions

Check the stuck PIDs' state (ps -o stat,wchan -p <pid>) — D state means storage/IO is hung; fix or reboot the affected storage
Wait for the IO to clear and retry the allocation; consider node drain/reboot if pids are permanently unkillable
Inspect dmesg for hung-task or filesystem errors explaining why SIGKILL was not honored
Ensure no leftover processes from prior allocations hold the cgroup (stop the leaking workload/driver)

Defensive patterns

Strategy: retry

Validate before calling

pids, err := cgroups.GetAllPids(filepath.Join(cgroupslib.GetDefaultRoot(), cgPath))
if err == nil && len(pids) > 0 {
    // orphans present: kill and confirm they leave D-state before Launch
    for _, pid := range pids {
        b, _ := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
        fmt.Printf("pid %d stat: %s\n", pid, b)
    }
}

Try / catch

if err := executor.Launch(cmd); err != nil && strings.Contains(err.Error(), "have not been removed from cgroups pid file") {
    // check D-state/IO hang; consider retrying allocation after storage recovers
    log.Printf("orphans stuck: %v", err)
}

Prevention

Monitor node storage health — D-state processes are unkillable until IO returns
Avoid NFS/FUSE inside task cgroups where SIGKILL cannot interrupt IO
Drain and reboot nodes with persistently unkillable orphans

When it happens

Trigger: orphanedPIDs remain in the cgroup's procs/pid file after the retry loop even though SIGKILL was sent — typically processes stuck in uninterruptible (D) state or unkillable kernel threads/IO waits.

Common situations: Orphaned process blocked in NFS/FUSE I/O so SIGKILL cannot complete; zombie processes parented elsewhere; storage (especially network filesystems) hung on the node; cgroup freezing or pid-file inconsistencies.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c215a3ed66671aef. Report an issue: GitHub.