{"record":{"id":"0bacab154a6c3c6a","repo":"hashicorp/nomad","slug":"unable-to-get-orphaned-task-pids-v","errorCode":null,"errorMessage":"unable to get orphaned task PIDs: %v","messagePattern":"unable to get orphaned task PIDs: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/shared/executor/executor_linux_cgo.go","lineNumber":148,"sourceCode":"\tgo le.catchSignals()\n\n\tle.processStats = procstats.New(compute, le)\n\treturn le\n}\n\nfunc (l *LibcontainerExecutor) ListProcesses() set.Collection[int] {\n\treturn procstats.List(l.command)\n}\n\n// cleanOldProcessesInCGroup kills processes that might ended up orphans when\n// the executor was unexpectedly killed and nomad can't reconnect to them.\nfunc (l *LibcontainerExecutor) cleanOldProcessesInCGroup(nomadRelativePath string) error {\n\tl.logger.Debug(\"looking for old processes\", \"path\", nomadRelativePath)\n\n\troot := cgroupslib.GetDefaultRoot()\n\torphanedPIDs, err := cgroups.GetAllPids(filepath.Join(root, nomadRelativePath))\n\tif err != nil && !os.IsNotExist(err) {\n\t\treturn fmt.Errorf(\"unable to get orphaned task PIDs: %v\", err)\n\t}\n\n\tfor _, pid := range orphanedPIDs {\n\t\tl.logger.Info(\"killing orphaned process\", \"pid\", pid)\n\n\t\t// Avoid bringing down the whole node by mistake, very unlikely case,\n\t\t// but it's better to be sure.\n\t\tif pid == 1 {\n\t\t\tcontinue\n\t\t}\n\n\t\tif err := syscall.Kill(pid, syscall.SIGKILL); err != nil {\n\t\t\treturn fmt.Errorf(\"unable to send signal to process %d: %v\", pid, err)\n\t\t}\n\t}\n\n\tif len(orphanedPIDs) == 0 {\n\t\treturn nil","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/shared/executor/executor_linux_cgo.go#L130-L166","documentation":"cleanOldProcessesInCGroup reads cgroup.procs via cgroups.GetAllPids to find orphaned processes left in the task's cgroup from a previous launch. If reading the PIDs fails and the error is NOT os.IsNotExist (i.e. the cgroup dir exists but its pid files are unreadable/broken), it returns this error. Missing cgroup is tolerated; unreadable cgroup is not.","triggerScenarios":"cgroups.GetAllPids(filepath.Join(cgroupslib.GetDefaultRoot(), nomadRelativePath)) fails with an error other than file-not-exist: permission denied on cgroup.procs, malformed cgroup filesystem, or I/O error reading the pid file.","commonSituations":"Cgroup mount permissions tightened (e.g. cgroup v2 paths restricted to root); the cgroup path exists but is corrupt or partially deleted; Nomad's cgroups root path misconfigured (nomad data dir vs /sys/fs/cgroup mismatch); kernel/cgroup driver changed between Nomad restarts.","solutions":["Inspect the cgroup path (cat <root>/<nomadRelativePath>/cgroup.procs) as the Nomad user to reproduce the read failure","Fix mount/ownership permissions on the cgroup hierarchy used by Nomad (v1 vs v2, systemd driver)","Verify nomad's cgroups_v2 configuration and that GetDefaultRoot matches the actually mounted hierarchy","Restart nomad to recreate the cgroup structure if the path is stale/corrupt"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"pidFile := filepath.Join(cgroupslib.GetDefaultRoot(), nomadRelativePath, \"cgroup.procs\")\nif _, err := os.Stat(pidFile); err == nil {\n    if f, err := os.Open(pidFile); err != nil {\n        // cgroup exists but unreadable: fix perms before Launch\n        _ = f\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := executor.Launch(cmd); err != nil && strings.Contains(err.Error(), \"unable to get orphaned task PIDs\") {\n    // inspect cgroup perms / mount, repair or recreate cgroup, then retry once\n    log.Printf(\"cgroup pids unreadable: %v\", err)\n}","preventionTips":["Verify cgroup mounts and permissions as the Nomad user before starting clients","Keep cgroup driver (v1/v2) config consistent across client restarts","Monitor for stale cgroup directories under the Nomad cgroup root"],"tags":["linux","cgroups","pids","executor"],"backgroundTag":"cgroup-pids-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}