What causes "wait for verified orphan backend pid %d: %w" in gastownhall/beads?

Wait a few seconds and confirm the pid is gone (ps -p / Get-Process -Id ), then re-run cleanup.

wait for verified orphan backend pid %d: %w

Error message

wait for verified orphan backend pid %d: %w

What it means

The orphan backend was killed and its handle closed; cleanup then waits up to backendExitTimeout for the recorded PID to actually exit (waitForRecordedProcessExit). This error means the process did not confirm exit within the timeout, so the record is not quarantined and cleanup aborts to avoid racing a still-dying process.

Source

Thrown at internal/storage/dbproxy/proxy/endpoint.go:846

	}
	if dead {
		if _, err := quarantineRecord(rootDir, server.PIDFileName, time.Now()); err != nil {
			return fmt.Errorf("quarantine dead backend record: %w", err)
		}
		return nil
	}
	if err := handle.Kill(); err != nil {
		_ = handle.Close()
		return fmt.Errorf("kill verified orphan backend pid %d from %s: %w", pf.Pid, recordPath, err)
	}
	// Close before waiting: an open handle on Windows keeps the dead PID
	// allocated, so procid.Verify would keep matching it and the exit wait
	// below would always time out.
	if err := handle.Close(); err != nil {
		return fmt.Errorf("close verified backend process handle for pid %d: %w", pf.Pid, err)
	}
	if err := waitForRecordedProcessExit(pf, backendExitTimeout); err != nil {
		return fmt.Errorf("wait for verified orphan backend pid %d: %w", pf.Pid, err)
	}
	if _, err := quarantineRecord(rootDir, server.PIDFileName, time.Now()); err != nil {
		return fmt.Errorf("quarantine orphan backend record: %w", err)
	}
	return nil
}

type unverifiableProcessChecks struct {
	LiveEstablished bool
	LegacyProxy     bool
}

func unverifiableProcessError(
	operation string,
	recordPath string,
	pid int,
	cause error,
	checks unverifiableProcessChecks,

View on GitHub (pinned to 71377f2769)

Solutions

Wait a few seconds and confirm the pid is gone (ps -p <pid> / Get-Process -Id <pid>), then re-run cleanup.
If the process is in uninterruptible sleep, resolve the blocking I/O (storage, NFS) or reboot the host.
Manually quarantine once the pid is confirmed dead: mv <recordPath> <recordPath>.stale-<unix-ts>.
Run cleanup in the same PID namespace / container context as the backend so exit verification works.

Example fix

// before
Error: wait for verified orphan backend pid 4242: timed out waiting for exit
// after
$ sleep 5 && ps -p 4242    # confirm gone
$ mv /ws/.beads/dbproxy/dolt-backend.pid /ws/.beads/dbproxy/dolt-backend.pid.stale-$(date +%s)

Defensive patterns

Strategy: retry

Try / catch

if err := cleanupOrphanBackend(rootDir); err != nil && strings.Contains(err.Error(), "wait for verified orphan backend pid") {
    // wait for exit out-of-band, then retry cleanup
    for i := 0; i < 30 && pidAlive(pfPid); i++ { time.Sleep(time.Second) }
    if !pidAlive(pfPid) {
        os.Rename(recordPath, recordPath+".stale-"+strconv.FormatInt(time.Now().Unix(), 10))
    }
}

Prevention

Ensure backend storage is local/fast so processes exit promptly after SIGKILL.
Run cleanup in the same PID namespace as the backend (containers).
Avoid NFS-backed workspace roots for pidfiles.
If a process sits in D state, resolve the underlying I/O stall before retrying.

When it happens

Trigger: handle.Kill() and handle.Close() succeeded, but waitForRecordedProcessExit(pf, backendExitTimeout) returns an error at internal/storage/dbproxy/proxy/endpoint.go:845-846 — the pid remained live (or verification kept matching it) past the timeout.

Common situations: Process stuck in uninterruptible sleep (D state) ignoring SIGKILL; child threads/zombie reaping delays; NFS/container environments where pid liveness checks misbehave; an extremely slow disk flush delaying exit.

Related errors

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/200e80b740657350. Report an issue: GitHub.