{"record":{"id":"8b8d67965072444c","repo":"neondatabase/neon","slug":"timed-out-waiting-for-postgres-to-start","errorCode":null,"errorMessage":"Timed out waiting for postgres to start","messagePattern":"Timed out waiting for postgres to start","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"control_plane/src/storage_controller.rs","lineNumber":468,"sourceCode":"            ];\n            tracing::info!(\n                \"Starting storage controller database with args: {:?}\",\n                db_start_args\n            );\n\n            let db_start_status = self.pg_ctl(db_start_args).await;\n            let start_timeout: Duration = start_args.start_timeout.into();\n            let db_start_deadline = Instant::now() + start_timeout;\n            if !db_start_status.success() {\n                return Err(anyhow::anyhow!(\n                    \"Failed to start postgres {}\",\n                    db_start_status.code().unwrap()\n                ));\n            }\n\n            loop {\n                if Instant::now() > db_start_deadline {\n                    return Err(anyhow::anyhow!(\"Timed out waiting for postgres to start\"));\n                }\n\n                match self.pg_isready(&pg_bin_dir, postgres_port).await {\n                    Ok(true) => {\n                        tracing::info!(\"storage controller postgres is now ready\");\n                        break;\n                    }\n                    Ok(false) => {\n                        tokio::time::sleep(Duration::from_millis(100)).await;\n                    }\n                    Err(e) => {\n                        tracing::warn!(\"Failed to check postgres status: {e}\")\n                    }\n                }\n            }\n\n            self.setup_database(postgres_port).await?;\n        }","sourceCodeStart":450,"sourceCodeEnd":486,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/storage_controller.rs#L450-L486","documentation":"After pg_ctl start returns success, storcon polls pg_isready on the database port until start_timeout elapses. If postgres never reports ready before the deadline, this timeout is raised; pg_isready errors are only logged as warnings while the loop keeps polling.","triggerScenarios":"postgres is slow to become ready (cold cache, heavy fsync on a slow disk), postgres listens on a different port than the one polled, or it crashes after pg_ctl already returned success.","commonSituations":"Scale tests and CI runners with slow local disks, very small start_timeout values, port mixups between the dynamically chosen postgres_port and the port written to postgresql.conf.","solutions":["Raise the start timeout (start_timeout / --start-timeout) for slow environments","Check storage_controller_db/log/ to see whether postgres eventually started or died","Verify the polled port matches the port configured in postgresql.conf","Once postgres is confirmed up, retry the storcon start — the state is recoverable"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"let timeout: std::time::Duration = start_args.start_timeout.into();\nanyhow::ensure!(timeout >= std::time::Duration::from_secs(10), \"start_timeout too small for this machine\");","typeGuard":null,"tryCatchPattern":"match self.pg_isready(&pg_bin_dir, postgres_port).await {\n    Ok(true) => break,\n    Ok(false) if Instant::now() < deadline => { tokio::time::sleep(Duration::from_millis(100)).await; }\n    Ok(false) => { /* raise --start-timeout and retry the start */ }\n    Err(e) => { tracing::warn!(\"isready failed: {e}\"); /* keep polling until deadline */ }\n}","preventionTips":["Bump start timeout on slow CI disks","Check the pg log before assuming a hang","Confirm the polled port equals the port written to postgresql.conf"],"tags":["rust","storage-controller","postgres","timeout","startup"],"backgroundTag":"service-startup-timeout","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}