shutdown stage daemon.process_reap: daemon did not exit afte

Error message

shutdown stage daemon.process_reap: daemon did not exit after forced termination; the singleton lock may still be held

What it means

After a shutdown is acknowledged but the process survives past the grace window, the CLI escalates with identity-gated SIGTERM then SIGKILL. If even after forced termination the PID is still alive (KillOutcome::StillAlive), the CLI cannot confirm exit and warns that the singleton state-db lock may still be held; it deliberately leaves runtime markers in place so start/restart see an actionable state instead of a raw lock error.

Source

Thrown at crates/astrid-cli/src/commands/daemon.rs:837

    eprintln!(
        "{}",
        theme::Theme::warning(
            "Daemon acknowledged shutdown but is still running; escalating with a signal so the \
             state-db lock is released."
        )
    );
    let outcome = daemon_control::terminate_identity(&identity, pid_path).await;
    confirm_kill_outcome(outcome)
}

fn confirm_kill_outcome(outcome: daemon_control::KillOutcome) -> Result<DaemonStopDisposition> {
    match outcome {
        daemon_control::KillOutcome::NotRunning => Ok(DaemonStopDisposition::AlreadyStopped),
        daemon_control::KillOutcome::TermExited | daemon_control::KillOutcome::KilledExited => {
            Ok(DaemonStopDisposition::Forced)
        },
        daemon_control::KillOutcome::StillAlive => {
            anyhow::bail!(
                "shutdown stage daemon.process_reap: daemon did not exit after forced termination; the singleton lock may still be held"
            );
        },
        daemon_control::KillOutcome::Unverified(pid) => {
            anyhow::bail!(
                "shutdown stage daemon.process_identity: PID {pid} is live but cannot be verified as Astrid; no markers were removed"
            );
        },
    }
}

/// Fence marker cleanup with the same singleton lock the daemon owns. Holding
/// it through removal prevents a replacement daemon from publishing fresh
/// markers between liveness proof and cleanup.
async fn cleanup_daemon_runtime(socket_path: &Path, pid_path: &Path) -> Result<()> {
    let home = astrid_core::dirs::AstridHome::resolve()
        .context("shutdown stage daemon.home_resolution")?;
    cleanup_daemon_runtime_for_home(&home, socket_path, pid_path).await

View on GitHub (pinned to affd8760f4)

Solutions

Check the process state (ps -o stat= -p <pid>); if it is in D state, resolve the blocked I/O (unmount/repair the hung filesystem).
Wait and re-run `astrid stop` — an unreaped but dead process will clear as its parent reaps it.
Reboot or (in containers) restart the container/namespace if the process is genuinely unkillable.
Once the PID is gone, remove leftover markers or run `astrid restart` to clean and respawn.

Example fix

// before
Error: shutdown stage daemon.process_reap: daemon did not exit after forced termination; the singleton lock may still be held
// after
$ ps -o pid,stat,wchan -p <pid>   # inspect why it will not die
$ # resolve blocked I/O or restart the container/host namespace, then:
$ astrid restart

Defensive patterns

Strategy: retry

Validate before calling

// check whether the recorded PID is in uninterruptible sleep before retrying
fn pid_stuck_in_d_state(pid: i32) -> bool {
    std::fs::read_to_string(format!("/proc/{pid}/stat"))
        .map(|s| s.rsplit_once(' ').map_or(false, |(_, _)| s.contains(" D ")))
        .unwrap_or(false)
}

Type guard

null

Try / catch

if let Err(e) = astrid_stop().await {
    if e.to_string().contains("did not exit after forced termination") {
        eprintln!("daemon unkillable; resolve blocked I/O or restart the host/container, then retry");
        // wait for the PID to disappear, then retry once
        wait_pid_gone(recorded_pid, GRACE).await;
        astrid_stop().await?;
    } else { return Err(e); }
}

Prevention

Avoid network (NFS/FUSE) mounts for the state DB or logs — D-state hangs make SIGKILL ineffective.
Monitor for daemons stuck mid-shutdown and alert before automation retries.
Use `astrid restart` for wedged daemons instead of stacking manual signals.
If this recurs on a host, check kernel/storage health — user-space signals cannot reap D-state tasks.

When it happens

Trigger: confirm_kill_outcome receives KillOutcome::StillAlive: terminate_identity sent TERM and KILL but wait checks still find the PID alive within the allowed window — a truly wedged/uninterruptible (D-state) process.

Common situations: Daemon stuck in uninterruptible I/O (NFS/fuse hang, dead disk) so even SIGKILL can't reap it; kernel-level hangs; PID stuck in a zombie state due to a stuck parent; container PID namespace weirdness.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/42083411c88528dd. Report an issue: GitHub.