{"record":{"id":"c215a3ed66671aef","repo":"hashicorp/nomad","slug":"orphaned-processes-v-have-not-been-removed-from-c","errorCode":null,"errorMessage":"orphaned processes %v have not been removed from cgroups pid file","messagePattern":"orphaned processes (.+?) have not been removed from cgroups pid file","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/shared/executor/executor_linux_cgo.go","lineNumber":180,"sourceCode":"\t\t}\n\t}\n\n\tif len(orphanedPIDs) == 0 {\n\t\treturn nil\n\t}\n\n\t// Make sure the PID was removed from the cgroup file, otherwise\n\t// libcontainer will not be able to launch. Five retries every 100 ms should be\n\t// more than enough.\n\tfor i := 100; i < 501; i += 100 {\n\t\torphanedPIDs, _ = cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))\n\t\tif len(orphanedPIDs) > 0 {\n\t\t\ttime.Sleep(time.Duration(i) * time.Millisecond)\n\t\t\tcontinue\n\t\t}\n\t\treturn nil\n\t}\n\treturn fmt.Errorf(\"orphaned processes %v have not been removed from cgroups pid file\", orphanedPIDs)\n}\n\n// Launch creates a new container in libcontainer and starts a new process with it\nfunc (l *LibcontainerExecutor) Launch(command *ExecCommand) (*ProcessState, error) {\n\tl.logger.Trace(\"preparing to launch command\", \"command\", command.Cmd, \"args\", strings.Join(command.Args, \" \"))\n\n\tif command.Resources == nil {\n\t\tcommand.Resources = &drivers.Resources{\n\t\t\tNomadResources: &structs.AllocatedTaskResources{},\n\t\t}\n\t}\n\n\tl.command = command\n\n\t// A container groups processes under the same isolation enforcement\n\tcontainerCfg, err := l.newLibcontainerConfig(command)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to configure container(%s): %v\", l.id, err)","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/shared/executor/executor_linux_cgo.go#L162-L198","documentation":"After sending SIGKILL to orphaned PIDs, the function polls (up to ~10 iterations with backoff sleeps) waiting for the PIDs to disappear from the cgroup pid file. If they are still present after the retries, it returns this error, refusing to launch a new task into a cgroup that still holds live processes.","triggerScenarios":"orphanedPIDs remain in the cgroup's procs/pid file after the retry loop even though SIGKILL was sent — typically processes stuck in uninterruptible (D) state or unkillable kernel threads/IO waits.","commonSituations":"Orphaned process blocked in NFS/FUSE I/O so SIGKILL cannot complete; zombie processes parented elsewhere; storage (especially network filesystems) hung on the node; cgroup freezing or pid-file inconsistencies.","solutions":["Check the stuck PIDs' state (ps -o stat,wchan -p <pid>) — D state means storage/IO is hung; fix or reboot the affected storage","Wait for the IO to clear and retry the allocation; consider node drain/reboot if pids are permanently unkillable","Inspect dmesg for hung-task or filesystem errors explaining why SIGKILL was not honored","Ensure no leftover processes from prior allocations hold the cgroup (stop the leaking workload/driver)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"pids, err := cgroups.GetAllPids(filepath.Join(cgroupslib.GetDefaultRoot(), cgPath))\nif err == nil && len(pids) > 0 {\n    // orphans present: kill and confirm they leave D-state before Launch\n    for _, pid := range pids {\n        b, _ := os.ReadFile(fmt.Sprintf(\"/proc/%d/stat\", pid))\n        fmt.Printf(\"pid %d stat: %s\\n\", pid, b)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := executor.Launch(cmd); err != nil && strings.Contains(err.Error(), \"have not been removed from cgroups pid file\") {\n    // check D-state/IO hang; consider retrying allocation after storage recovers\n    log.Printf(\"orphans stuck: %v\", err)\n}","preventionTips":["Monitor node storage health — D-state processes are unkillable until IO returns","Avoid NFS/FUSE inside task cgroups where SIGKILL cannot interrupt IO","Drain and reboot nodes with persistently unkillable orphans"],"tags":["linux","cgroups","process-lifecycle","executor"],"backgroundTag":"orphaned-processes-not-reaped","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}