{"record":{"id":"68fbe16e4ecc052b","repo":"gastownhall/beads","slug":"procid-process-d-still-matches-token-after-fatal-68fbe1","errorCode":null,"errorMessage":"procid: process %d still matches token after fatal signal and %s re-check","messagePattern":"procid: process (.+?) still matches token after fatal signal and (.+?) re-check","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/procid/procid_linux.go","lineNumber":178,"sourceCode":"\t}\n\tif !match {\n\t\treturn fmt.Errorf(\"procid: process %d no longer matches token\", h.pid)\n\t}\n\treturn nil\n}\n\nfunc (h *Handle) confirmFatalSignal() error {\n\tdeadline := time.Now().Add(fallbackSignalConfirmTimeout)\n\tfor {\n\t\tmatch, err := Verify(h.pid, h.token)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif !match {\n\t\t\treturn nil\n\t\t}\n\t\tif time.Now().After(deadline) {\n\t\t\treturn fmt.Errorf(\n\t\t\t\t\"procid: process %d still matches token after fatal signal and %s re-check\",\n\t\t\t\th.pid,\n\t\t\t\tfallbackSignalConfirmTimeout,\n\t\t\t)\n\t\t}\n\t\ttime.Sleep(10 * time.Millisecond)\n\t}\n}\n\nfunc isFatalSignal(sig syscall.Signal) bool {\n\treturn sig == syscall.SIGKILL || sig == syscall.SIGTERM\n}\n\nfunc processStartTime(pid int) (string, error) {\n\tdata, err := os.ReadFile(\"/proc/\" + strconv.Itoa(pid) + \"/stat\")\n\tif err != nil {\n\t\treturn \"\", &processStatReadError{pid: pid, err: err}\n\t}","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/procid/procid_linux.go#L160-L196","documentation":"This error is returned by confirmFatalSignal in the Linux procid implementation after sending a fatal signal and re-checking whether the process still matches the captured identity token (PID + start time). It means the target process refused to die (or the signal did not take effect) within the fallbackSignalConfirmTimeout deadline, so procid cannot confirm the kill and refuses to report success to avoid PID-reuse ambiguity.","triggerScenarios":"Calling a procid kill/confirm path (confirmFatalSignal) when the process is stuck in uninterruptible sleep (D state), ignores or blocks the fatal signal (SIGKILL should not, but pre-confirmation re-checks may race), the PID was reused by a different process whose token no longer matches yet still resolves as matching due to stale /proc data, or the 10ms-poll loop exhausts the deadline before the kernel finishes reaping the process.","commonSituations":"Killing a wedged worker stuck in unkillable I/O (NFS, FUSE, blocked driver); signaling a process in a cgroup frozen state; heavily loaded machines where process teardown is slow; containers with frozen cgroups (SIGKILL deferred); PID namespaces where the reaper is slow.","solutions":["Increase fallbackSignalConfirmTimeout if your workload legitimately tears down slowly","Check the target process state in /proc/<pid>/stat for 'D' (uninterruptible) state and resolve the underlying I/O block (e.g. unstick NFS/FUSE mount)","Verify the process is not in a frozen cgroup (check cgroup.freeze / freezer controller) and thaw it before signaling","Re-capture the token and retry the confirm loop; if the PID was reused, the token re-check will fail cleanly and the caller can treat the old process as gone"],"exampleFix":"// before\nif err := procid.KillConfirmed(pid, token); err != nil {\n\treturn fmt.Errorf(\"worker did not die: %w\", err)\n}\n// after\nif err := procid.KillConfirmed(pid, token); err != nil {\n\tvar perr *procid.StillAliveError\n\tif errors.As(err, &perr) {\n\t\t// inspect /proc/<pid>/stat state; thaw cgroup or escalate\n\t}\n\treturn fmt.Errorf(\"worker did not die: %w\", err)\n}","handlingStrategy":"retry","validationCode":"// before issuing the fatal signal, confirm the process is killable\nfunc killable(pid int) bool {\n\tb, err := os.ReadFile(fmt.Sprintf(\"/proc/%d/stat\", pid))\n\tif err != nil {\n\t\treturn false\n\t}\n\ti := strings.LastIndexByte(string(b), ')')\n\tif i < 0 || len(b) <= i+2 {\n\t\treturn false\n\t}\n\tstate := rune(b[i+2])\n\treturn state != 'Z' && state != 'X' && state != 'x' // 'D' state will likely time out\n}","typeGuard":null,"tryCatchPattern":"err := procid.KillConfirmed(pid, tok)\nif errors.Is(err, procid.ErrStillAlive) { // or match on message/timeout\n\t// one bounded retry after checking process state, then escalate\n\ttime.Sleep(500 * time.Millisecond)\n\terr = procid.KillConfirmed(pid, tok)\n}\nif err != nil {\n\tlog.Warnf(\"process %d not confirmed dead: %v\", pid, err)\n}","preventionTips":["Inspect /proc/<pid>/stat state for 'D' before signaling; resolve blocked I/O first","Ensure the target is not in a frozen cgroup (cgroup.freeze) before killing","Keep fallbackSignalConfirmTimeout generous on slow/loaded hosts","Always pass the captured token so PID reuse is detected instead of killing an innocent new process"],"tags":["linux","process-management","timeout","signal"],"backgroundTag":"process-kill-timeout","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}