{"record":{"id":"ff5e78fd8b3064be","repo":"neondatabase/neon","slug":"pg-ctl-failed-exit-code-stdout-stderr","errorCode":null,"errorMessage":"pg_ctl failed, exit code: {}, stdout: {}, stderr: {}","messagePattern":"pg_ctl failed, exit code: (.+?), stdout: (.+?), stderr: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"control_plane/src/endpoint.rs","lineNumber":631,"sourceCode":"        .env(\n            \"LD_LIBRARY_PATH\",\n            self.env.pg_lib_dir(self.pg_version)?.to_str().unwrap(),\n        )\n        .env(\n            \"DYLD_LIBRARY_PATH\",\n            self.env.pg_lib_dir(self.pg_version)?.to_str().unwrap(),\n        );\n\n        // Pass authentication token used for the connections to pageserver and safekeepers\n        if let Some(token) = auth_token {\n            cmd.env(\"NEON_AUTH_TOKEN\", token);\n        }\n\n        let pg_ctl = cmd\n            .output()\n            .context(format!(\"{} failed\", pg_ctl_path.display()))?;\n        if !pg_ctl.status.success() {\n            anyhow::bail!(\n                \"pg_ctl failed, exit code: {}, stdout: {}, stderr: {}\",\n                pg_ctl.status,\n                String::from_utf8_lossy(&pg_ctl.stdout),\n                String::from_utf8_lossy(&pg_ctl.stderr),\n            );\n        }\n\n        Ok(())\n    }\n\n    fn wait_for_compute_ctl_to_exit(&self, send_sigterm: bool) -> Result<()> {\n        // TODO use background_process::stop_process instead: https://github.com/neondatabase/neon/pull/6482\n        let pidfile_path = self.endpoint_path().join(\"compute_ctl.pid\");\n        let pid: u32 = std::fs::read_to_string(pidfile_path)?.parse()?;\n        let pid = nix::unistd::Pid::from_raw(pid as i32);\n        if send_sigterm {\n            kill(pid, Signal::SIGTERM).ok();\n        }","sourceCodeStart":613,"sourceCodeEnd":649,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/endpoint.rs#L613-L649","documentation":"Endpoint::pg_ctl shells out to the Postgres pg_ctl binary (path built from env.pg_lib_dir(pg_version)); a non-zero exit of pg_ctl raises this error including exit code, stdout and stderr. pg_ctl failing means Postgres itself refused the operation — the embedded stderr almost always contains pg_ctl's 'FATAL'/hint lines that pinpoint the cause. Compare this with spawn failures, which surface earlier as a `{pg_ctl_path} failed` context error when the binary can't be executed at all.","triggerScenarios":"Calling Endpoint::start (pg_ctl start/wait), or stop with non-immediate modes (pg_ctl -m fast stop). Fails when: the port is already bound, postgresql.conf/generated config has an invalid setting, pgdata is inconsistent, the selected pg_lib_dir doesn't match the pgdata version, or pg_ctl times out waiting for startup.","commonSituations":"Leftover postgres from a previous unclean run holding the port; version mismatch after switching pg version in the env while reusing an old endpoint dir; bad settings in shared postgresql.conf snippets (e.g. invalid extension GUC); missing pg binaries at the expected target/pg/install path after a rebuild moved them; disk full preventing pid file creation.","solutions":["Read the stderr field in the message — the actual FATAL line (e.g. 'could not bind', 'permission denied', 'setting not recognized') tells you the fix directly.","Free the conflicting port: find the holder with `lsof -i :<port>` (or `ss -ltnp`) and stop the stale process / run `neon_local endpoint stop`.","If the config is at fault, fix the offending setting in the endpoint's postgresql.conf or your shared config, and retry start.","If it's a version mismatch, recreate the endpoint (`neon_local endpoint destroy` + create) against the current pg_lib_dir, or point the env at the matching pg build.","Check `env.pg_lib_dir(pg_version)` exists and contains bin/pg_ctl for the requested version."],"exampleFix":"// before\nself.pg_ctl(&[\"-l\", log_path, \"start\", \"-w\"], &None)?; // exit code 1, stderr hidden until printed\n\n// after\nif let Err(e) = self.pg_ctl(&[\"-l\", log_path, \"start\", \"-w\"], &None) {\n    let tail = std::fs::read_to_string(self.pgdata().join(\"log/startup.log\"))\n        .unwrap_or_default();\n    anyhow::bail!(\"{e:#}\\n--- pg startup log ---\\n{tail}\");\n}","handlingStrategy":"try-catch","validationCode":"// pre-start checks that remove the most common pg_ctl failure modes\nanyhow::ensure!(TcpListener::bind((\"127.0.0.1\", port)).is_ok(), \"port {port} already in use\");\nanyhow::ensure!(env.pg_lib_dir(pg_version)?.join(\"bin/pg_ctl\").exists(), \"pg_ctl missing in pg_lib_dir\");","typeGuard":null,"tryCatchPattern":"match endpoint.pg_ctl(&[\"-m\", \"fast\", \"stop\"], &None) {\n    Err(e) => {\n        // message embeds pg_ctl stderr; surface it plus the startup log\n        let log = std::fs::read_to_string(endpoint.pgdata().join(\"log/startup.log\")).unwrap_or_default();\n        tracing::error!(\"{e:#}\\n{log}\");\n        Err(e)\n    }\n    ok => ok,\n}","preventionTips":["Free ports before starting endpoints; run `neon_local env stop` in teardown.","Recreate endpoints after changing the postgres version in the env.","Keep postgresql.conf snippets valid — bad GUCs fail pg_ctl before status checks run.","After rebuilds, verify target/pg/install/<ver>/bin/pg_ctl exists for every version you use."],"tags":["rust","neon","control-plane","endpoint","postgres","pg-ctl","subprocess"],"backgroundTag":"pg-ctl-startup-failure","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}