{"record":{"id":"a0f7d676b890cf24","repo":"gastownhall/beads","slug":"proxy-forcestopunverified-timeout-waiting-for-pid","errorCode":null,"errorMessage":"proxy.ForceStopUnverified: timeout waiting for pid %d to exit","messagePattern":"proxy\\.ForceStopUnverified: timeout waiting for pid (.+?) to exit","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/dbproxy/proxy/force_stop.go","lineNumber":274,"sourceCode":"\tif err != nil {\n\t\treturn fmt.Errorf(\"proxy.ForceStopUnverified: signal pid %d: %w\", pid, err)\n\t}\n\tif gone {\n\t\treport.ProcessWasGone = true\n\t\treturn nil\n\t}\n\treport.SignalSent = true\n\n\tfor {\n\t\texited, err := proc.exited()\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"proxy.ForceStopUnverified: confirm pid %d exit: %w\", pid, err)\n\t\t}\n\t\tif exited {\n\t\t\treturn nil\n\t\t}\n\t\tif time.Now().After(deadline) {\n\t\t\treturn fmt.Errorf(\"proxy.ForceStopUnverified: timeout waiting for pid %d to exit\", pid)\n\t\t}\n\t\ttime.Sleep(shutdownConfirmPoll)\n\t}\n}\n\nfunc normalizeForceStopExecutable(name string) string {\n\tname = strings.TrimSpace(filepath.Base(name))\n\tname = strings.TrimSuffix(name, \" (deleted)\")\n\tname = strings.TrimSuffix(strings.ToLower(name), \".exe\")\n\treturn name\n}\n\nfunc acquireForceStopLock(lockPath string, deadline time.Time) (*util.Lock, error) {\n\tfor {\n\t\tlock, err := util.TryLock(lockPath)\n\t\tif err == nil {\n\t\t\treturn lock, nil\n\t\t}","sourceCodeStart":256,"sourceCodeEnd":292,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/storage/dbproxy/proxy/force_stop.go#L256-L292","documentation":"This error is returned by inspectAndStopUnverifiedPID when a signaled (SIGKILL'd) unverified process did not confirm exit before the ForceStopUnverified deadline elapsed. Since SIGKILL cannot be caught by a process, this indicates the kernel has not finished tearing the process down within the timeout, or the poll loop kept observing it as alive. It signals that force-stop could not fully verify cleanup even after signaling.","triggerScenarios":"Call ForceStopUnverified (`bd dolt stop --force`) on a live record PID that passes verification, gets killed (report.SignalSent=true), yet proc.exited() still returns false when time.Now() passes the deadline (default shutdownConfirmDeadline or the ForceStopOptions.Timeout you passed).","commonSituations":"Very short custom ForceStopOptions.Timeout values that do not allow kernel process teardown; heavily loaded machines or high load average slowing exit finalization; processes stuck in uninterruptible kernel states (D state, e.g. blocked I/O on NFS/Dolt storage) that cannot die promptly even on SIGKILL.","solutions":["Wait a few seconds and re-run `bd dolt stop --force`; the process will typically be reaped and take the ProcessWasGone path","Increase ForceStopOptions.Timeout (or accept the default) to give SIGKILL teardown time to complete","Check the process state with `ps -o stat= -p <pid>`; if it is in D state, resolve the blocked I/O (NFS hang, disk issue) so the kernel can finish the exit","If the process truly will not die, investigate kernel/systemd-oomd/parent reaping, then quarantine the record manually (rename to *.stale-<unix-timestamp>) once the PID is confirmed gone"],"exampleFix":"// before\nreport, err := proxy.ForceStopUnverified(rootDir, proxy.ForceStopOptions{Timeout: 2 * time.Second})\n// after\nreport, err := proxy.ForceStopUnverified(rootDir, proxy.ForceStopOptions{Timeout: 30 * time.Second})","handlingStrategy":"retry","validationCode":"// Before force-stop, check the process state so D-state (uninterruptible) processes are detected:\nout, _ := os.ReadFile(fmt.Sprintf(\"/proc/%d/stat\", pid))\nstate := strings.Fields(string(out))[2] // \"D\" means stuck in uninterruptible I/O","typeGuard":null,"tryCatchPattern":"report, err := proxy.ForceStopUnverified(rootDir, proxy.ForceStopOptions{Timeout: 30 * time.Second})\nif err != nil && strings.Contains(err.Error(), \"timeout waiting for pid\") {\n    // Wait for kernel teardown, then retry; second run usually sees ProcessWasGone\n    time.Sleep(5 * time.Second)\n    report, err = proxy.ForceStopUnverified(rootDir)\n}","preventionTips":["Do not shrink ForceStopOptions.Timeout below a few seconds; SIGKILL teardown can be slow under load","Avoid force-stopping processes blocked on dead storage (NFS/fuse); resolve the I/O hang first","Inspect report.SignalSent to distinguish 'kill never sent' from 'kill sent but exit unconfirmed'","After this error, verify with ps that the PID is actually gone before quarantining the record manually"],"tags":["process-management","go","timeout","force-stop"],"backgroundTag":"process-kill-timeout","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}