{"record":{"id":"75c64214020491d6","repo":"neondatabase/neon","slug":"failed-to-start-postgres","errorCode":null,"errorMessage":"Failed to start postgres {}","messagePattern":"Failed to start postgres (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"control_plane/src/storage_controller.rs","lineNumber":460,"sourceCode":"                \"-w\",\n                \"-D\",\n                pg_data_path.as_ref(),\n                \"-l\",\n                pg_log_path.as_ref(),\n                \"-U\",\n                &username(),\n                \"start\",\n            ];\n            tracing::info!(\n                \"Starting storage controller database with args: {:?}\",\n                db_start_args\n            );\n\n            let db_start_status = self.pg_ctl(db_start_args).await;\n            let start_timeout: Duration = start_args.start_timeout.into();\n            let db_start_deadline = Instant::now() + start_timeout;\n            if !db_start_status.success() {\n                return Err(anyhow::anyhow!(\n                    \"Failed to start postgres {}\",\n                    db_start_status.code().unwrap()\n                ));\n            }\n\n            loop {\n                if Instant::now() > db_start_deadline {\n                    return Err(anyhow::anyhow!(\"Timed out waiting for postgres to start\"));\n                }\n\n                match self.pg_isready(&pg_bin_dir, postgres_port).await {\n                    Ok(true) => {\n                        tracing::info!(\"storage controller postgres is now ready\");\n                        break;\n                    }\n                    Ok(false) => {\n                        tokio::time::sleep(Duration::from_millis(100)).await;\n                    }","sourceCodeStart":442,"sourceCodeEnd":478,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/storage_controller.rs#L442-L478","documentation":"storcon starts its private postgres via pg_ctl -D <pgdata> start with the storage controller user; a non-success pg_ctl exit fails immediately with the numeric exit code in the message. Note the message prints db_start_status.code().unwrap(), so a pg_ctl terminated by a signal would panic instead of producing this error.","triggerScenarios":"The postgres port is already in use, postgresql.conf is invalid, the pgdata dir is missing or owned by another user, a stale postmaster.pid blocks startup, or required libs/binaries are missing.","commonSituations":"Port collisions between concurrent test runs, leftover postmaster.pid after a killed env, permission mismatch when storcon runs under a different user than the one that created the data dir.","solutions":["Read the postgres server log under storage_controller_db/log/ — it contains the real startup failure","Free the port or let the env pick another; remove a stale postmaster.pid from the pgdata dir after confirming the pid is dead","Fix ownership/permissions of storage_controller_db to the running user","Re-init the database directory (drop storage_controller_db) if the cluster is corrupted"],"exampleFix":"// before\nreturn Err(anyhow::anyhow!(\"Failed to start postgres {}\", db_start_status.code().unwrap()));\n// after — no panic when pg_ctl was killed by a signal\nreturn Err(anyhow::anyhow!(\"Failed to start postgres, exit code: {:?}\", db_start_status.code()));","handlingStrategy":"retry","validationCode":"// before pg_ctl start: rule out the common causes\nlet port_free = std::net::TcpListener::bind((\"127.0.0.1\", postgres_port)).is_ok();\nanyhow::ensure!(port_free, \"postgres port {postgres_port} already in use\");\nlet pid_file = pg_data_path.join(\"postmaster.pid\");\nif pid_file.exists() { /* verify the recorded pid is dead, then remove the file */ }","typeGuard":null,"tryCatchPattern":"match self.pg_ctl(db_start_args).await.code() {\n    Some(0) => {}\n    Some(code) => { /* read <pgdata>/log/* for the cause; fix port/pid/conf; retry start */ }\n    None => { /* pg_ctl killed by a signal: check dmesg for OOM */ }\n}","preventionTips":["Check dynamic ports are free before starting storcon's postgres","Remove postmaster.pid only after confirming the pid is dead","Run storcon and its postgres as the same user"],"tags":["rust","storage-controller","postgres","pg-ctl","startup"],"backgroundTag":"postgres-start-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}