{"record":{"id":"200e80b740657350","repo":"gastownhall/beads","slug":"wait-for-verified-orphan-backend-pid-d-w","errorCode":null,"errorMessage":"wait for verified orphan backend pid %d: %w","messagePattern":"wait for verified orphan backend pid (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/dbproxy/proxy/endpoint.go","lineNumber":846,"sourceCode":"\t}\n\tif dead {\n\t\tif _, err := quarantineRecord(rootDir, server.PIDFileName, time.Now()); err != nil {\n\t\t\treturn fmt.Errorf(\"quarantine dead backend record: %w\", err)\n\t\t}\n\t\treturn nil\n\t}\n\tif err := handle.Kill(); err != nil {\n\t\t_ = handle.Close()\n\t\treturn fmt.Errorf(\"kill verified orphan backend pid %d from %s: %w\", pf.Pid, recordPath, err)\n\t}\n\t// Close before waiting: an open handle on Windows keeps the dead PID\n\t// allocated, so procid.Verify would keep matching it and the exit wait\n\t// below would always time out.\n\tif err := handle.Close(); err != nil {\n\t\treturn fmt.Errorf(\"close verified backend process handle for pid %d: %w\", pf.Pid, err)\n\t}\n\tif err := waitForRecordedProcessExit(pf, backendExitTimeout); err != nil {\n\t\treturn fmt.Errorf(\"wait for verified orphan backend pid %d: %w\", pf.Pid, err)\n\t}\n\tif _, err := quarantineRecord(rootDir, server.PIDFileName, time.Now()); err != nil {\n\t\treturn fmt.Errorf(\"quarantine orphan backend record: %w\", err)\n\t}\n\treturn nil\n}\n\ntype unverifiableProcessChecks struct {\n\tLiveEstablished bool\n\tLegacyProxy     bool\n}\n\nfunc unverifiableProcessError(\n\toperation string,\n\trecordPath string,\n\tpid int,\n\tcause error,\n\tchecks unverifiableProcessChecks,","sourceCodeStart":828,"sourceCodeEnd":864,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/storage/dbproxy/proxy/endpoint.go#L828-L864","documentation":"The orphan backend was killed and its handle closed; cleanup then waits up to backendExitTimeout for the recorded PID to actually exit (waitForRecordedProcessExit). This error means the process did not confirm exit within the timeout, so the record is not quarantined and cleanup aborts to avoid racing a still-dying process.","triggerScenarios":"handle.Kill() and handle.Close() succeeded, but waitForRecordedProcessExit(pf, backendExitTimeout) returns an error at internal/storage/dbproxy/proxy/endpoint.go:845-846 — the pid remained live (or verification kept matching it) past the timeout.","commonSituations":"Process stuck in uninterruptible sleep (D state) ignoring SIGKILL; child threads/zombie reaping delays; NFS/container environments where pid liveness checks misbehave; an extremely slow disk flush delaying exit.","solutions":["Wait a few seconds and confirm the pid is gone (ps -p <pid> / Get-Process -Id <pid>), then re-run cleanup.","If the process is in uninterruptible sleep, resolve the blocking I/O (storage, NFS) or reboot the host.","Manually quarantine once the pid is confirmed dead: mv <recordPath> <recordPath>.stale-<unix-ts>.","Run cleanup in the same PID namespace / container context as the backend so exit verification works."],"exampleFix":"// before\nError: wait for verified orphan backend pid 4242: timed out waiting for exit\n// after\n$ sleep 5 && ps -p 4242    # confirm gone\n$ mv /ws/.beads/dbproxy/dolt-backend.pid /ws/.beads/dbproxy/dolt-backend.pid.stale-$(date +%s)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := cleanupOrphanBackend(rootDir); err != nil && strings.Contains(err.Error(), \"wait for verified orphan backend pid\") {\n    // wait for exit out-of-band, then retry cleanup\n    for i := 0; i < 30 && pidAlive(pfPid); i++ { time.Sleep(time.Second) }\n    if !pidAlive(pfPid) {\n        os.Rename(recordPath, recordPath+\".stale-\"+strconv.FormatInt(time.Now().Unix(), 10))\n    }\n}","preventionTips":["Ensure backend storage is local/fast so processes exit promptly after SIGKILL.","Run cleanup in the same PID namespace as the backend (containers).","Avoid NFS-backed workspace roots for pidfiles.","If a process sits in D state, resolve the underlying I/O stall before retrying."],"tags":["go","process-exit","timeout","orphan-backend"],"backgroundTag":"process-exit-timeout","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}