What causes "pg_ctl status returned unexpected status code: {:?}" in neondatabase/neon?

Run pg_ctl -D /storage_controller_db status by hand and read its stderr — it explains the unexpected code

pg_ctl status returned unexpected status code: {:?}

Error message

pg_ctl status returned unexpected status code: {:?}

What it means

storcon classifies pg_ctl status exit codes: 0 = running, 3 = not running, 4 = no data dir. Any other exit code (permissions errors on the data directory, usage errors, differing pg_ctl semantics) is unexpected and reported with this error.

Source

Thrown at control_plane/src/storage_controller.rs:770

        Ok(())
    }

    async fn is_postgres_running(&self) -> anyhow::Result<bool> {
        let pg_data_path = self.env.base_data_dir.join("storage_controller_db");

        let pg_status_args = ["-D", &pg_data_path.to_string_lossy(), "status"];
        let status_exitcode = self.pg_ctl(pg_status_args).await;

        // pg_ctl status returns this exit code if postgres is not running: in this case it is
        // fine that stop failed.  Otherwise it is an error that stop failed.
        const PG_STATUS_NOT_RUNNING: i32 = 3;
        const PG_NO_DATA_DIR: i32 = 4;
        const PG_STATUS_RUNNING: i32 = 0;
        match status_exitcode.code() {
            Some(PG_STATUS_NOT_RUNNING) => Ok(false),
            Some(PG_NO_DATA_DIR) => Ok(false),
            Some(PG_STATUS_RUNNING) => Ok(true),
            Some(code) => Err(anyhow::anyhow!(
                "pg_ctl status returned unexpected status code: {:?}",
                code
            )),
            None => Err(anyhow::anyhow!("pg_ctl status returned no status code")),
        }
    }

    fn get_claims_for_path(path: &str) -> anyhow::Result<Option<Claims>> {
        let category = match path.find('/') {
            Some(idx) => &path[..idx],
            None => path,
        };

        match category {
            "status" | "ready" => Ok(None),
            "control" | "debug" => Ok(Some(Claims::new(None, Scope::Admin))),
            "v1" => Ok(Some(Claims::new(None, Scope::PageServerApi))),
            _ => Err(anyhow::anyhow!("Failed to determine claims for {}", path)),

View on GitHub (pinned to 8f60b04da4)

Solutions

Run pg_ctl -D <base>/storage_controller_db status by hand and read its stderr — it explains the unexpected code
Fix ownership/permissions on the data dir
Rebuild or reinstall the postgres distribution if pg_ctl itself is broken

Defensive patterns

Strategy: validation

Validate before calling

let code = pg_ctl(["-D", &dir, "status"]).await.code();
match code {
    Some(0) | Some(3) | Some(4) => { /* classified */ }
    Some(c) => tracing::warn!("unexpected pg_ctl status code {c}; run it manually for stderr"),
    None => tracing::warn!("pg_ctl terminated by a signal"),
}

Prevention

Do not chmod/chown the data dir between runs
Pin one postgres distribution per env
Wrap pg_ctl calls to capture stderr so unexpected codes are diagnosable

When it happens

Trigger: pg_ctl status exits with an out-of-set code: inaccessible or wrongly owned pgdata, an older/newer pg_ctl with different behavior, or a partially deleted data dir that pg_ctl cannot inspect.

Common situations: Running as a different user than the owner of storage_controller_db, mixing postgres versions under pg_install, partially deleted data directories.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/0bcc349acad8e9db. Report an issue: GitHub.