hashicorp/nomad · error

unable to send signal to process %d: %v

Error message

unable to send signal to process %d: %v

What it means

After collecting orphaned PIDs, cleanOldProcessesInCGroup kills each with syscall.Kill(pid, SIGKILL), skipping PID 1 to protect the node. If the kill syscall fails for any orphan (e.g. the process disappeared before the signal, or permission denied), this error aborts the launch. It exists to prevent reaping wrong/stale processes silently.

Source

Thrown at drivers/shared/executor/executor_linux_cgo.go:161

	l.logger.Debug("looking for old processes", "path", nomadRelativePath)

	root := cgroupslib.GetDefaultRoot()
	orphanedPIDs, err := cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))
	if err != nil && !os.IsNotExist(err) {
		return fmt.Errorf("unable to get orphaned task PIDs: %v", err)
	}

	for _, pid := range orphanedPIDs {
		l.logger.Info("killing orphaned process", "pid", pid)

		// Avoid bringing down the whole node by mistake, very unlikely case,
		// but it's better to be sure.
		if pid == 1 {
			continue
		}

		if err := syscall.Kill(pid, syscall.SIGKILL); err != nil {
			return fmt.Errorf("unable to send signal to process %d: %v", pid, err)
		}
	}

	if len(orphanedPIDs) == 0 {
		return nil
	}

	// Make sure the PID was removed from the cgroup file, otherwise
	// libcontainer will not be able to launch. Five retries every 100 ms should be
	// more than enough.
	for i := 100; i < 501; i += 100 {
		orphanedPIDs, _ = cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))
		if len(orphanedPIDs) > 0 {
			time.Sleep(time.Duration(i) * time.Millisecond)
			continue
		}
		return nil
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Treat ESRCH as benign: upgrade Nomad or patch the check so syscall.ESRCH does not abort the launch
2. Ensure the Nomad/executor process has CAP_KILL and is not blocked by LSM (SELinux/AppArmor) policies
3. Verify the stale cgroup pid file contents (ps -p <pid>) — a PID belonging to another workload indicates a stale cgroup.procs
4. Recycle the client/cgroup to clear stale orphan entries

// before launching, check for orphans and confirm they are killable
pids, err := cgroups.GetAllPids(filepath.Join(cgroupslib.GetDefaultRoot(), cgPath))
if err == nil {
    for _, pid := range pids {
        if pid == 1 { continue }
        if err := syscall.Kill(pid, 0); err == syscall.ESRCH { /* stale, fine */ }
    }
}

if err := executor.Launch(cmd); err != nil && strings.Contains(err.Error(), "unable to send signal to process") {
    log.Printf("orphan kill failed: %v — check CAP_KILL/ESRCH race", err)
}

When it happens

Trigger: syscall.Kill(pid, syscall.SIGKILL) returns an error for a non-init orphaned PID: ESRCH (process already exited — the common case) or EPERM (kill not permitted for the executor user).

Common situations: Orphan exited between listing PIDs and killing them (race, ESRCH); cgroup pid file contained a stale PID belonging to another (now re-parented) process; running Nomad in a container without CAP_KILL; SELinux/AppArmor blocking signals.

Related errors

• ErrCgroupMustBeSet
• unable to get orphaned task PIDs: %v
• orphaned processes %v have not been removed from cgroups pid
• failed to set cpuset: %w
• failed to create nomad cgroup %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/25016ffcabd0280b. Report an issue: GitHub.