What causes "process exited with {status}" in neondatabase/neon?

Treat non-zero exit as 'not ready yet': retry with a deadline instead of failing on the first attempt

process exited with {status}

Error message

process exited with {status}

What it means

pg_isready(bin, port) spawned `<bin> -p <port>` and the child exited non-zero: the Postgres server is not accepting connections right now. pg_isready conventionally exits 1 when the server is running but rejecting connections (starting up or shutting down) and 2 when no connection could be made (nothing listening / wrong port). The error is wrapped with 'could not run `{bin} --port {port}`' and, in the Hadron liveness probe route, becomes an HTTP 500 response.

Source

Thrown at compute_tools/src/pg_isready.rs:19

use anyhow::{Context, anyhow};

// Run `/usr/local/bin/pg_isready -p {port}`
// Check the connectivity of PG
// Success means PG is listening on the port and accepting connections
// Note that PG does not need to authenticate the connection, nor reserve a connection quota for it.
// See https://www.postgresql.org/docs/current/app-pg-isready.html
pub fn pg_isready(bin: &str, port: u16) -> anyhow::Result<()> {
    let child_result = std::process::Command::new(bin)
        .arg("-p")
        .arg(port.to_string())
        .spawn();

    child_result
        .context("spawn() failed")
        .and_then(|mut child| child.wait().context("wait() failed"))
        .and_then(|status| match status.success() {
            true => Ok(()),
            false => Err(anyhow!("process exited with {status}")),
        })
        // wrap any prior error with the overall context that we couldn't run the command
        .with_context(|| format!("could not run `{bin} --port {port}`"))
}

// It's safe to assume pg_isready is under the same directory with postgres,
// because it is a PG util bin installed along with postgres
pub fn get_pg_isready_bin(pgbin: &str) -> String {
    let split = pgbin.split("/").collect::<Vec<&str>>();
    split[0..split.len() - 1].join("/") + "/pg_isready"
}

View on GitHub (pinned to 8f60b04da4)

Solutions

Treat non-zero exit as 'not ready yet': retry with a deadline instead of failing on the first attempt
Verify a listener exists: `ss -ltnp | grep <port>` and inspect Postgres logs for startup/crash errors
Confirm the pg_isready path - it is derived from the pgbin directory (get_pg_isready_bin)
Use the exit code to route debugging: 'exit code: 1' = server explicitly rejecting (still starting); 'exit code: 2' = connect failed (wrong port or down)

Example fix

// before
pg_isready(bin, port)?;

// after
let deadline = Instant::now() + Duration::from_secs(30);
loop {
    match pg_isready(bin, port) {
        Ok(()) => break,
        Err(e) if Instant::now() < deadline => thread::sleep(Duration::from_millis(500)),
        Err(e) => return Err(e),
    }
}

Defensive patterns

Strategy: retry

Validate before calling

// cheap pre-check: is anything listening before spawning pg_isready?
use std::net::TcpStream;
let listening = TcpStream::connect(("127.0.0.1", port)).is_ok();

Try / catch

// readiness loop instead of one-shot
let deadline = Instant::now() + Duration::from_secs(30);
loop {
    match pg_isready(bin, port) {
        Ok(()) => break,
        Err(e) if Instant::now() < deadline => thread::sleep(Duration::from_millis(500)),
        Err(e) => return Err(e),
    }
}

Prevention

Gate first traffic with a readiness loop, not a single pg_isready call
Derive the probe port from the same connstr the server listens on
Keep pg_isready in the same directory as the postgres binary (get_pg_isready_bin assumption)
Use the child's exit code to distinguish 'still starting' from 'nothing listening'

When it happens

Trigger: The liveness probe (hadron_liveness_probe) runs before Postgres finished startup (exit 1); the port taken from the compute connstr has nothing listening (exit 2); Postgres crashed; connection-level failure (firewall, socket). A missing binary would fail at spawn() instead with a different message.

Common situations: Kubernetes liveness/readiness probes firing during slow cold starts, basebackup restore, or recovery; misconfigured connstr port; pg_isready binary not located next to the postgres binary (get_pg_isready_bin assumes same directory).

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/acde6d4c12a32d6c. Report an issue: GitHub.