{"record":{"id":"25016ffcabd0280b","repo":"hashicorp/nomad","slug":"unable-to-send-signal-to-process-d-v","errorCode":null,"errorMessage":"unable to send signal to process %d: %v","messagePattern":"unable to send signal to process (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/shared/executor/executor_linux_cgo.go","lineNumber":161,"sourceCode":"\tl.logger.Debug(\"looking for old processes\", \"path\", nomadRelativePath)\n\n\troot := cgroupslib.GetDefaultRoot()\n\torphanedPIDs, err := cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))\n\tif err != nil && !os.IsNotExist(err) {\n\t\treturn fmt.Errorf(\"unable to get orphaned task PIDs: %v\", err)\n\t}\n\n\tfor _, pid := range orphanedPIDs {\n\t\tl.logger.Info(\"killing orphaned process\", \"pid\", pid)\n\n\t\t// Avoid bringing down the whole node by mistake, very unlikely case,\n\t\t// but it's better to be sure.\n\t\tif pid == 1 {\n\t\t\tcontinue\n\t\t}\n\n\t\tif err := syscall.Kill(pid, syscall.SIGKILL); err != nil {\n\t\t\treturn fmt.Errorf(\"unable to send signal to process %d: %v\", pid, err)\n\t\t}\n\t}\n\n\tif len(orphanedPIDs) == 0 {\n\t\treturn nil\n\t}\n\n\t// Make sure the PID was removed from the cgroup file, otherwise\n\t// libcontainer will not be able to launch. Five retries every 100 ms should be\n\t// more than enough.\n\tfor i := 100; i < 501; i += 100 {\n\t\torphanedPIDs, _ = cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))\n\t\tif len(orphanedPIDs) > 0 {\n\t\t\ttime.Sleep(time.Duration(i) * time.Millisecond)\n\t\t\tcontinue\n\t\t}\n\t\treturn nil\n\t}","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/shared/executor/executor_linux_cgo.go#L143-L179","documentation":"After collecting orphaned PIDs, cleanOldProcessesInCGroup kills each with syscall.Kill(pid, SIGKILL), skipping PID 1 to protect the node. If the kill syscall fails for any orphan (e.g. the process disappeared before the signal, or permission denied), this error aborts the launch. It exists to prevent reaping wrong/stale processes silently.","triggerScenarios":"syscall.Kill(pid, syscall.SIGKILL) returns an error for a non-init orphaned PID: ESRCH (process already exited — the common case) or EPERM (kill not permitted for the executor user).","commonSituations":"Orphan exited between listing PIDs and killing them (race, ESRCH); cgroup pid file contained a stale PID belonging to another (now re-parented) process; running Nomad in a container without CAP_KILL; SELinux/AppArmor blocking signals.","solutions":["Treat ESRCH as benign: upgrade Nomad or patch the check so syscall.ESRCH does not abort the launch","Ensure the Nomad/executor process has CAP_KILL and is not blocked by LSM (SELinux/AppArmor) policies","Verify the stale cgroup pid file contents (ps -p <pid>) — a PID belonging to another workload indicates a stale cgroup.procs","Recycle the client/cgroup to clear stale orphan entries"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// before launching, check for orphans and confirm they are killable\npids, err := cgroups.GetAllPids(filepath.Join(cgroupslib.GetDefaultRoot(), cgPath))\nif err == nil {\n    for _, pid := range pids {\n        if pid == 1 { continue }\n        if err := syscall.Kill(pid, 0); err == syscall.ESRCH { /* stale, fine */ }\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := executor.Launch(cmd); err != nil && strings.Contains(err.Error(), \"unable to send signal to process\") {\n    log.Printf(\"orphan kill failed: %v — check CAP_KILL/ESRCH race\", err)\n}","preventionTips":["Run Nomad with CAP_KILL and without restrictive SELinux/AppArmor signal rules","Expect ESRCH races for short-lived orphans; upgrade to a Nomad version that tolerates them","Audit stale cgroup.procs entries for PIDs reassigned to other workloads"],"tags":["linux","signals","kill","cgroups","executor"],"backgroundTag":"kill-signal-permission-denied","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}