proxy.ForceStopUnverified: timeout waiting for pid %d to exi

Error message

proxy.ForceStopUnverified: timeout waiting for pid %d to exit

What it means

This error is returned by inspectAndStopUnverifiedPID when a signaled (SIGKILL'd) unverified process did not confirm exit before the ForceStopUnverified deadline elapsed. Since SIGKILL cannot be caught by a process, this indicates the kernel has not finished tearing the process down within the timeout, or the poll loop kept observing it as alive. It signals that force-stop could not fully verify cleanup even after signaling.

Source

Thrown at internal/storage/dbproxy/proxy/force_stop.go:274

	if err != nil {
		return fmt.Errorf("proxy.ForceStopUnverified: signal pid %d: %w", pid, err)
	}
	if gone {
		report.ProcessWasGone = true
		return nil
	}
	report.SignalSent = true

	for {
		exited, err := proc.exited()
		if err != nil {
			return fmt.Errorf("proxy.ForceStopUnverified: confirm pid %d exit: %w", pid, err)
		}
		if exited {
			return nil
		}
		if time.Now().After(deadline) {
			return fmt.Errorf("proxy.ForceStopUnverified: timeout waiting for pid %d to exit", pid)
		}
		time.Sleep(shutdownConfirmPoll)
	}
}

func normalizeForceStopExecutable(name string) string {
	name = strings.TrimSpace(filepath.Base(name))
	name = strings.TrimSuffix(name, " (deleted)")
	name = strings.TrimSuffix(strings.ToLower(name), ".exe")
	return name
}

func acquireForceStopLock(lockPath string, deadline time.Time) (*util.Lock, error) {
	for {
		lock, err := util.TryLock(lockPath)
		if err == nil {
			return lock, nil
		}

View on GitHub (pinned to 71377f2769)

Solutions

Wait a few seconds and re-run `bd dolt stop --force`; the process will typically be reaped and take the ProcessWasGone path
Increase ForceStopOptions.Timeout (or accept the default) to give SIGKILL teardown time to complete
Check the process state with `ps -o stat= -p <pid>`; if it is in D state, resolve the blocked I/O (NFS hang, disk issue) so the kernel can finish the exit
If the process truly will not die, investigate kernel/systemd-oomd/parent reaping, then quarantine the record manually (rename to *.stale-<unix-timestamp>) once the PID is confirmed gone

Example fix

// before
report, err := proxy.ForceStopUnverified(rootDir, proxy.ForceStopOptions{Timeout: 2 * time.Second})
// after
report, err := proxy.ForceStopUnverified(rootDir, proxy.ForceStopOptions{Timeout: 30 * time.Second})

Defensive patterns

Strategy: retry

Validate before calling

// Before force-stop, check the process state so D-state (uninterruptible) processes are detected:
out, _ := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
state := strings.Fields(string(out))[2] // "D" means stuck in uninterruptible I/O

Try / catch

report, err := proxy.ForceStopUnverified(rootDir, proxy.ForceStopOptions{Timeout: 30 * time.Second})
if err != nil && strings.Contains(err.Error(), "timeout waiting for pid") {
    // Wait for kernel teardown, then retry; second run usually sees ProcessWasGone
    time.Sleep(5 * time.Second)
    report, err = proxy.ForceStopUnverified(rootDir)
}

Prevention

Do not shrink ForceStopOptions.Timeout below a few seconds; SIGKILL teardown can be slow under load
Avoid force-stopping processes blocked on dead storage (NFS/fuse); resolve the I/O hang first
Inspect report.SignalSent to distinguish 'kill never sent' from 'kill sent but exit unconfirmed'
After this error, verify with ps that the PID is actually gone before quarantining the record manually

When it happens

Trigger: Call ForceStopUnverified (`bd dolt stop --force`) on a live record PID that passes verification, gets killed (report.SignalSent=true), yet proc.exited() still returns false when time.Now() passes the deadline (default shutdownConfirmDeadline or the ForceStopOptions.Timeout you passed).

Common situations: Very short custom ForceStopOptions.Timeout values that do not allow kernel process teardown; heavily loaded machines or high load average slowing exit finalization; processes stuck in uninterruptible kernel states (D state, e.g. blocked I/O on NFS/Dolt storage) that cannot die promptly even on SIGKILL.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/a0f7d676b890cf24. Report an issue: GitHub.