neondatabase/neon · error · anyhow::Error

pg_ctl failed, exit code: {}, stdout: {}, stderr: {}

Error message

pg_ctl failed, exit code: {}, stdout: {}, stderr: {}

What it means

Endpoint::pg_ctl shells out to the Postgres pg_ctl binary (path built from env.pg_lib_dir(pg_version)); a non-zero exit of pg_ctl raises this error including exit code, stdout and stderr. pg_ctl failing means Postgres itself refused the operation — the embedded stderr almost always contains pg_ctl's 'FATAL'/hint lines that pinpoint the cause. Compare this with spawn failures, which surface earlier as a `{pg_ctl_path} failed` context error when the binary can't be executed at all.

Source

Thrown at control_plane/src/endpoint.rs:631

        .env(
            "LD_LIBRARY_PATH",
            self.env.pg_lib_dir(self.pg_version)?.to_str().unwrap(),
        )
        .env(
            "DYLD_LIBRARY_PATH",
            self.env.pg_lib_dir(self.pg_version)?.to_str().unwrap(),
        );

        // Pass authentication token used for the connections to pageserver and safekeepers
        if let Some(token) = auth_token {
            cmd.env("NEON_AUTH_TOKEN", token);
        }

        let pg_ctl = cmd
            .output()
            .context(format!("{} failed", pg_ctl_path.display()))?;
        if !pg_ctl.status.success() {
            anyhow::bail!(
                "pg_ctl failed, exit code: {}, stdout: {}, stderr: {}",
                pg_ctl.status,
                String::from_utf8_lossy(&pg_ctl.stdout),
                String::from_utf8_lossy(&pg_ctl.stderr),
            );
        }

        Ok(())
    }

    fn wait_for_compute_ctl_to_exit(&self, send_sigterm: bool) -> Result<()> {
        // TODO use background_process::stop_process instead: https://github.com/neondatabase/neon/pull/6482
        let pidfile_path = self.endpoint_path().join("compute_ctl.pid");
        let pid: u32 = std::fs::read_to_string(pidfile_path)?.parse()?;
        let pid = nix::unistd::Pid::from_raw(pid as i32);
        if send_sigterm {
            kill(pid, Signal::SIGTERM).ok();
        }

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Read the stderr field in the message — the actual FATAL line (e.g. 'could not bind', 'permission denied', 'setting not recognized') tells you the fix directly.
2. Free the conflicting port: find the holder with `lsof -i :<port>` (or `ss -ltnp`) and stop the stale process / run `neon_local endpoint stop`.
3. If the config is at fault, fix the offending setting in the endpoint's postgresql.conf or your shared config, and retry start.
4. If it's a version mismatch, recreate the endpoint (`neon_local endpoint destroy` + create) against the current pg_lib_dir, or point the env at the matching pg build.
5. Check `env.pg_lib_dir(pg_version)` exists and contains bin/pg_ctl for the requested version.

Example fix

// before
self.pg_ctl(&["-l", log_path, "start", "-w"], &None)?; // exit code 1, stderr hidden until printed

// after
if let Err(e) = self.pg_ctl(&["-l", log_path, "start", "-w"], &None) {
    let tail = std::fs::read_to_string(self.pgdata().join("log/startup.log"))
        .unwrap_or_default();
    anyhow::bail!("{e:#}\n--- pg startup log ---\n{tail}");
}

// pre-start checks that remove the most common pg_ctl failure modes
anyhow::ensure!(TcpListener::bind(("127.0.0.1", port)).is_ok(), "port {port} already in use");
anyhow::ensure!(env.pg_lib_dir(pg_version)?.join("bin/pg_ctl").exists(), "pg_ctl missing in pg_lib_dir");

match endpoint.pg_ctl(&["-m", "fast", "stop"], &None) {
    Err(e) => {
        // message embeds pg_ctl stderr; surface it plus the startup log
        let log = std::fs::read_to_string(endpoint.pgdata().join("log/startup.log")).unwrap_or_default();
        tracing::error!("{e:#}\n{log}");
        Err(e)
    }
    ok => ok,
}

When it happens

Trigger: Calling Endpoint::start (pg_ctl start/wait), or stop with non-immediate modes (pg_ctl -m fast stop). Fails when: the port is already bound, postgresql.conf/generated config has an invalid setting, pgdata is inconsistent, the selected pg_lib_dir doesn't match the pgdata version, or pg_ctl times out waiting for startup.

Common situations: Leftover postgres from a previous unclean run holding the port; version mismatch after switching pg version in the env while reusing an old endpoint dir; bad settings in shared postgresql.conf snippets (e.g. invalid extension GUC); missing pg binaries at the expected target/pg/install path after a rebuild moved them; disk full preventing pid file creation.

Related errors

• Endpoint::from_dir_entry failed: '{}' is not a directory
• safekeeper {sk_id} does not exist
• postgres --sync-safekeepers exited with non-zero status: {}.
• process exited with {status}
• `datadir` must be a directory when calling this function: {d

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/ff5e78fd8b3064be. Report an issue: GitHub.