{"record":{"id":"42083411c88528dd","repo":"astrid-runtime/astrid","slug":"shutdown-stage-daemon-process-reap-daemon-did-not","errorCode":null,"errorMessage":"shutdown stage daemon.process_reap: daemon did not exit after forced termination; the singleton lock may still be held","messagePattern":"shutdown stage daemon\\.process_reap: daemon did not exit after forced termination; the singleton lock may still be held","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/astrid-cli/src/commands/daemon.rs","lineNumber":837,"sourceCode":"    eprintln!(\n        \"{}\",\n        theme::Theme::warning(\n            \"Daemon acknowledged shutdown but is still running; escalating with a signal so the \\\n             state-db lock is released.\"\n        )\n    );\n    let outcome = daemon_control::terminate_identity(&identity, pid_path).await;\n    confirm_kill_outcome(outcome)\n}\n\nfn confirm_kill_outcome(outcome: daemon_control::KillOutcome) -> Result<DaemonStopDisposition> {\n    match outcome {\n        daemon_control::KillOutcome::NotRunning => Ok(DaemonStopDisposition::AlreadyStopped),\n        daemon_control::KillOutcome::TermExited | daemon_control::KillOutcome::KilledExited => {\n            Ok(DaemonStopDisposition::Forced)\n        },\n        daemon_control::KillOutcome::StillAlive => {\n            anyhow::bail!(\n                \"shutdown stage daemon.process_reap: daemon did not exit after forced termination; the singleton lock may still be held\"\n            );\n        },\n        daemon_control::KillOutcome::Unverified(pid) => {\n            anyhow::bail!(\n                \"shutdown stage daemon.process_identity: PID {pid} is live but cannot be verified as Astrid; no markers were removed\"\n            );\n        },\n    }\n}\n\n/// Fence marker cleanup with the same singleton lock the daemon owns. Holding\n/// it through removal prevents a replacement daemon from publishing fresh\n/// markers between liveness proof and cleanup.\nasync fn cleanup_daemon_runtime(socket_path: &Path, pid_path: &Path) -> Result<()> {\n    let home = astrid_core::dirs::AstridHome::resolve()\n        .context(\"shutdown stage daemon.home_resolution\")?;\n    cleanup_daemon_runtime_for_home(&home, socket_path, pid_path).await","sourceCodeStart":819,"sourceCodeEnd":855,"githubUrl":"https://github.com/astrid-runtime/astrid/blob/affd8760f44190dbdfbec23403f4c4b642c33112/crates/astrid-cli/src/commands/daemon.rs#L819-L855","documentation":"After a shutdown is acknowledged but the process survives past the grace window, the CLI escalates with identity-gated SIGTERM then SIGKILL. If even after forced termination the PID is still alive (KillOutcome::StillAlive), the CLI cannot confirm exit and warns that the singleton state-db lock may still be held; it deliberately leaves runtime markers in place so start/restart see an actionable state instead of a raw lock error.","triggerScenarios":"confirm_kill_outcome receives KillOutcome::StillAlive: terminate_identity sent TERM and KILL but wait checks still find the PID alive within the allowed window — a truly wedged/uninterruptible (D-state) process.","commonSituations":"Daemon stuck in uninterruptible I/O (NFS/fuse hang, dead disk) so even SIGKILL can't reap it; kernel-level hangs; PID stuck in a zombie state due to a stuck parent; container PID namespace weirdness.","solutions":["Check the process state (ps -o stat= -p <pid>); if it is in D state, resolve the blocked I/O (unmount/repair the hung filesystem).","Wait and re-run `astrid stop` — an unreaped but dead process will clear as its parent reaps it.","Reboot or (in containers) restart the container/namespace if the process is genuinely unkillable.","Once the PID is gone, remove leftover markers or run `astrid restart` to clean and respawn."],"exampleFix":"// before\nError: shutdown stage daemon.process_reap: daemon did not exit after forced termination; the singleton lock may still be held\n// after\n$ ps -o pid,stat,wchan -p <pid>   # inspect why it will not die\n$ # resolve blocked I/O or restart the container/host namespace, then:\n$ astrid restart","handlingStrategy":"retry","validationCode":"// check whether the recorded PID is in uninterruptible sleep before retrying\nfn pid_stuck_in_d_state(pid: i32) -> bool {\n    std::fs::read_to_string(format!(\"/proc/{pid}/stat\"))\n        .map(|s| s.rsplit_once(' ').map_or(false, |(_, _)| s.contains(\" D \")))\n        .unwrap_or(false)\n}","typeGuard":"null","tryCatchPattern":"if let Err(e) = astrid_stop().await {\n    if e.to_string().contains(\"did not exit after forced termination\") {\n        eprintln!(\"daemon unkillable; resolve blocked I/O or restart the host/container, then retry\");\n        // wait for the PID to disappear, then retry once\n        wait_pid_gone(recorded_pid, GRACE).await;\n        astrid_stop().await?;\n    } else { return Err(e); }\n}","preventionTips":["Avoid network (NFS/FUSE) mounts for the state DB or logs — D-state hangs make SIGKILL ineffective.","Monitor for daemons stuck mid-shutdown and alert before automation retries.","Use `astrid restart` for wedged daemons instead of stacking manual signals.","If this recurs on a host, check kernel/storage health — user-space signals cannot reap D-state tasks."],"tags":["daemon","shutdown","signal","process"],"backgroundTag":"invalid-state-transition","analyzedSha":"affd8760f44190dbdfbec23403f4c4b642c33112","analyzedAt":"2026-09-09T21:28:12.402Z","contentChangedAt":"2026-09-09T21:28:12.402Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}