neondatabase/neon · error · anyhow::Error

Failed to start postgres {}

Error message

Failed to start postgres {}

What it means

storcon starts its private postgres via pg_ctl -D <pgdata> start with the storage controller user; a non-success pg_ctl exit fails immediately with the numeric exit code in the message. Note the message prints db_start_status.code().unwrap(), so a pg_ctl terminated by a signal would panic instead of producing this error.

Source

Thrown at control_plane/src/storage_controller.rs:460

                "-w",
                "-D",
                pg_data_path.as_ref(),
                "-l",
                pg_log_path.as_ref(),
                "-U",
                &username(),
                "start",
            ];
            tracing::info!(
                "Starting storage controller database with args: {:?}",
                db_start_args
            );

            let db_start_status = self.pg_ctl(db_start_args).await;
            let start_timeout: Duration = start_args.start_timeout.into();
            let db_start_deadline = Instant::now() + start_timeout;
            if !db_start_status.success() {
                return Err(anyhow::anyhow!(
                    "Failed to start postgres {}",
                    db_start_status.code().unwrap()
                ));
            }

            loop {
                if Instant::now() > db_start_deadline {
                    return Err(anyhow::anyhow!("Timed out waiting for postgres to start"));
                }

                match self.pg_isready(&pg_bin_dir, postgres_port).await {
                    Ok(true) => {
                        tracing::info!("storage controller postgres is now ready");
                        break;
                    }
                    Ok(false) => {
                        tokio::time::sleep(Duration::from_millis(100)).await;
                    }

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Read the postgres server log under storage_controller_db/log/ — it contains the real startup failure
2. Free the port or let the env pick another; remove a stale postmaster.pid from the pgdata dir after confirming the pid is dead
3. Fix ownership/permissions of storage_controller_db to the running user
4. Re-init the database directory (drop storage_controller_db) if the cluster is corrupted

Example fix

// before
return Err(anyhow::anyhow!("Failed to start postgres {}", db_start_status.code().unwrap()));
// after — no panic when pg_ctl was killed by a signal
return Err(anyhow::anyhow!("Failed to start postgres, exit code: {:?}", db_start_status.code()));

// before pg_ctl start: rule out the common causes
let port_free = std::net::TcpListener::bind(("127.0.0.1", postgres_port)).is_ok();
anyhow::ensure!(port_free, "postgres port {postgres_port} already in use");
let pid_file = pg_data_path.join("postmaster.pid");
if pid_file.exists() { /* verify the recorded pid is dead, then remove the file */ }

match self.pg_ctl(db_start_args).await.code() {
    Some(0) => {}
    Some(code) => { /* read <pgdata>/log/* for the cause; fix port/pid/conf; retry start */ }
    None => { /* pg_ctl killed by a signal: check dmesg for OOM */ }
}

When it happens

Trigger: The postgres port is already in use, postgresql.conf is invalid, the pgdata dir is missing or owned by another user, a stale postmaster.pid blocks startup, or required libs/binaries are missing.

Common situations: Port collisions between concurrent test runs, leftover postmaster.pid after a killed env, permission mismatch when storcon runs under a different user than the one that created the data dir.

Related errors

• Timed out waiting for postgres to start
• Failed to stop storage controller database
• Failed to stop storage controller database: {err}
• pg_ctl status returned unexpected status code: {:?}
• pg_ctl status returned no status code

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/75c64214020491d6. Report an issue: GitHub.