neondatabase/neon · error · anyhow::Error

Timed out waiting for postgres to start

Error message

Timed out waiting for postgres to start

What it means

After pg_ctl start returns success, storcon polls pg_isready on the database port until start_timeout elapses. If postgres never reports ready before the deadline, this timeout is raised; pg_isready errors are only logged as warnings while the loop keeps polling.

Source

Thrown at control_plane/src/storage_controller.rs:468

            ];
            tracing::info!(
                "Starting storage controller database with args: {:?}",
                db_start_args
            );

            let db_start_status = self.pg_ctl(db_start_args).await;
            let start_timeout: Duration = start_args.start_timeout.into();
            let db_start_deadline = Instant::now() + start_timeout;
            if !db_start_status.success() {
                return Err(anyhow::anyhow!(
                    "Failed to start postgres {}",
                    db_start_status.code().unwrap()
                ));
            }

            loop {
                if Instant::now() > db_start_deadline {
                    return Err(anyhow::anyhow!("Timed out waiting for postgres to start"));
                }

                match self.pg_isready(&pg_bin_dir, postgres_port).await {
                    Ok(true) => {
                        tracing::info!("storage controller postgres is now ready");
                        break;
                    }
                    Ok(false) => {
                        tokio::time::sleep(Duration::from_millis(100)).await;
                    }
                    Err(e) => {
                        tracing::warn!("Failed to check postgres status: {e}")
                    }
                }
            }

            self.setup_database(postgres_port).await?;
        }

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Raise the start timeout (start_timeout / --start-timeout) for slow environments
2. Check storage_controller_db/log/ to see whether postgres eventually started or died
3. Verify the polled port matches the port configured in postgresql.conf
4. Once postgres is confirmed up, retry the storcon start — the state is recoverable

let timeout: std::time::Duration = start_args.start_timeout.into();
anyhow::ensure!(timeout >= std::time::Duration::from_secs(10), "start_timeout too small for this machine");

match self.pg_isready(&pg_bin_dir, postgres_port).await {
    Ok(true) => break,
    Ok(false) if Instant::now() < deadline => { tokio::time::sleep(Duration::from_millis(100)).await; }
    Ok(false) => { /* raise --start-timeout and retry the start */ }
    Err(e) => { tracing::warn!("isready failed: {e}"); /* keep polling until deadline */ }
}

When it happens

Trigger: postgres is slow to become ready (cold cache, heavy fsync on a slow disk), postgres listens on a different port than the one polled, or it crashes after pg_ctl already returned success.

Common situations: Scale tests and CI runners with slow local disks, very small start_timeout values, port mixups between the dynamically chosen postgres_port and the port written to postgresql.conf.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to start postgres {}
• postgres --sync-safekeepers exited with non-zero status: {}.
• {} did not start+pass status checks within {:?} seconds
• Postgres directory '{}' not found in {}
• createdb failed with status {}: {stderr}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/8b8d67965072444c. Report an issue: GitHub.