{"record":{"id":"acde6d4c12a32d6c","repo":"neondatabase/neon","slug":"process-exited-with-status-acde6d","errorCode":null,"errorMessage":"process exited with {status}","messagePattern":"process exited with (.+?)","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"compute_tools/src/pg_isready.rs","lineNumber":19,"sourceCode":"use anyhow::{Context, anyhow};\n\n// Run `/usr/local/bin/pg_isready -p {port}`\n// Check the connectivity of PG\n// Success means PG is listening on the port and accepting connections\n// Note that PG does not need to authenticate the connection, nor reserve a connection quota for it.\n// See https://www.postgresql.org/docs/current/app-pg-isready.html\npub fn pg_isready(bin: &str, port: u16) -> anyhow::Result<()> {\n    let child_result = std::process::Command::new(bin)\n        .arg(\"-p\")\n        .arg(port.to_string())\n        .spawn();\n\n    child_result\n        .context(\"spawn() failed\")\n        .and_then(|mut child| child.wait().context(\"wait() failed\"))\n        .and_then(|status| match status.success() {\n            true => Ok(()),\n            false => Err(anyhow!(\"process exited with {status}\")),\n        })\n        // wrap any prior error with the overall context that we couldn't run the command\n        .with_context(|| format!(\"could not run `{bin} --port {port}`\"))\n}\n\n// It's safe to assume pg_isready is under the same directory with postgres,\n// because it is a PG util bin installed along with postgres\npub fn get_pg_isready_bin(pgbin: &str) -> String {\n    let split = pgbin.split(\"/\").collect::<Vec<&str>>();\n    split[0..split.len() - 1].join(\"/\") + \"/pg_isready\"\n}\n","sourceCodeStart":1,"sourceCodeEnd":31,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/compute_tools/src/pg_isready.rs#L1-L31","documentation":"pg_isready(bin, port) spawned `<bin> -p <port>` and the child exited non-zero: the Postgres server is not accepting connections right now. pg_isready conventionally exits 1 when the server is running but rejecting connections (starting up or shutting down) and 2 when no connection could be made (nothing listening / wrong port). The error is wrapped with 'could not run `{bin} --port {port}`' and, in the Hadron liveness probe route, becomes an HTTP 500 response.","triggerScenarios":"The liveness probe (hadron_liveness_probe) runs before Postgres finished startup (exit 1); the port taken from the compute connstr has nothing listening (exit 2); Postgres crashed; connection-level failure (firewall, socket). A missing binary would fail at spawn() instead with a different message.","commonSituations":"Kubernetes liveness/readiness probes firing during slow cold starts, basebackup restore, or recovery; misconfigured connstr port; pg_isready binary not located next to the postgres binary (get_pg_isready_bin assumes same directory).","solutions":["Treat non-zero exit as 'not ready yet': retry with a deadline instead of failing on the first attempt","Verify a listener exists: `ss -ltnp | grep <port>` and inspect Postgres logs for startup/crash errors","Confirm the pg_isready path - it is derived from the pgbin directory (get_pg_isready_bin)","Use the exit code to route debugging: 'exit code: 1' = server explicitly rejecting (still starting); 'exit code: 2' = connect failed (wrong port or down)"],"exampleFix":"// before\npg_isready(bin, port)?;\n\n// after\nlet deadline = Instant::now() + Duration::from_secs(30);\nloop {\n    match pg_isready(bin, port) {\n        Ok(()) => break,\n        Err(e) if Instant::now() < deadline => thread::sleep(Duration::from_millis(500)),\n        Err(e) => return Err(e),\n    }\n}","handlingStrategy":"retry","validationCode":"// cheap pre-check: is anything listening before spawning pg_isready?\nuse std::net::TcpStream;\nlet listening = TcpStream::connect((\"127.0.0.1\", port)).is_ok();","typeGuard":null,"tryCatchPattern":"// readiness loop instead of one-shot\nlet deadline = Instant::now() + Duration::from_secs(30);\nloop {\n    match pg_isready(bin, port) {\n        Ok(()) => break,\n        Err(e) if Instant::now() < deadline => thread::sleep(Duration::from_millis(500)),\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Gate first traffic with a readiness loop, not a single pg_isready call","Derive the probe port from the same connstr the server listens on","Keep pg_isready in the same directory as the postgres binary (get_pg_isready_bin assumption)","Use the child's exit code to distinguish 'still starting' from 'nothing listening'"],"tags":["postgres","pg-isready","health-check","process-exit","rust"],"backgroundTag":"postgres-not-accepting-connections","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}