{"record":{"id":"02576c0430a228f5","repo":"neondatabase/neon","slug":"did-not-start-pass-status-checks-within-se","errorCode":null,"errorMessage":"{} did not start+pass status checks within {:?} seconds","messagePattern":"(.+?) did not start\\+pass status checks within (.+?) seconds","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"control_plane/src/background_process.rs","lineNumber":163,"sourceCode":"                if retries == NOTICE_AFTER_RETRIES {\n                    // The process is taking a long time to start up. Keep waiting, but\n                    // print a message\n                    print!(\"\\n{process_name} has not started yet, continuing to wait\");\n                }\n                if retries % DOT_EVERY_RETRIES == 0 {\n                    print!(\".\");\n                    io::stdout().flush().unwrap();\n                }\n                tokio::time::sleep(RETRY_INTERVAL).await;\n            }\n            Err(e) => {\n                println!(\"error starting process {process_name:?}: {e:#}\");\n                return Err(e);\n            }\n        }\n    }\n    println!();\n    anyhow::bail!(format!(\n        \"{} did not start+pass status checks within {:?} seconds\",\n        process_name, retry_timeout\n    ));\n}\n\n/// Stops the process, using the pid file given. Returns Ok also if the process is already not running.\npub fn stop_process(\n    immediate: bool,\n    process_name: &str,\n    pid_file: &Utf8Path,\n) -> anyhow::Result<()> {\n    let pid = match pid_file::read(pid_file)\n        .with_context(|| format!(\"read pid_file {pid_file:?}\"))?\n    {\n        PidFileRead::NotExist => {\n            println!(\"{process_name} is already stopped: no pid file present at {pid_file:?}\");\n            return Ok(());\n        }","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/background_process.rs#L145-L181","documentation":"start_process polls the process's status check every RETRY_INTERVAL for up to retry_timeout seconds; if the process never both stays up and passes its status check within that window, it gives up with this error. The loop prints dots as it waits, and any immediate start failure (spawn error) is returned earlier as a distinct error. The process's own output is being appended to {datadir}/{process_name}.log, which is where the root cause usually is.","triggerScenarios":"Calling start_process for pageserver, safekeeper, or postgres when the binary crashes shortly after starting, when the status-check closure never returns true (health endpoint refuses connections, wrong port, auth token rejected), or when startup legitimately takes longer than retry_timeout (slow first-run page cache warmup, huge tenant basebackup).","commonSituations":"Port already in use by a leftover process from a previous run; corrupted or missing pageserver tenant data making postgres unable to connect to pageserver; misconfigured pg_hba/listen addresses so the status check can't reach the process; CI runners with slow I/O where Postgres startup exceeds the default timeout; stale pid files making the check report NotHeldByAnyProcess.","solutions":["Read {datadir}/{process_name}.log — the actual crash/startup error is at the tail of that file.","Verify no stale process holds the port: `lsof -i :<port>` or check the pid file, and kill leftovers or run `neon_local env stop`.","Run the process's status check manually (curl the health URL / connect to the port) with the same auth the closure uses.","Pass a larger retry_timeout/start_timeout to start_process (e.g. EndpointStartArgs.start_timeout) on slow machines.","Fix the underlying config error the log reveals (bad postgresql.conf setting, unreachable pageserver/safekeeper) and retry."],"exampleFix":"// before\nstart_process(..., process_name, datadir, retry_timeout, status_check).await?; // \"did not start+pass status checks within 30 seconds\"\n\n// after\nlet retry_timeout = retry_timeout.max(Duration::from_secs(120));\nmatch start_process(..., process_name, datadir, retry_timeout, status_check).await {\n    Err(e) => {\n        let log = std::fs::read_to_string(datadir.join(format!(\"{process_name}.log\")))\n            .unwrap_or_default();\n        anyhow::bail!(\"{e:#}\\n--- {process_name}.log tail ---\\n{}\", log.lines().rev().take(40).collect::<Vec<_>>().reverse().collect::<Vec<_>>().join(\"\\n\"));\n    }\n    ok => ok,\n}","handlingStrategy":"retry","validationCode":"// pre-flight: is the port free and the log path writable?\nlet log_path = datadir.join(format!(\"{process_name}.log\"));\nanyhow::ensure!(std::fs::OpenOptions::new().create(true).append(true).open(&log_path).is_ok(),\n    \"cannot write {log_path:?}\");\n// (port-binding is verified by the process itself; check via `ss` if flaky)","typeGuard":null,"tryCatchPattern":"match start_process(...).await {\n    Err(e) if e.to_string().contains(\"did not start+pass status checks\") => {\n        let log = std::fs::read_to_string(datadir.join(format!(\"{process_name}.log\"))).unwrap_or_default();\n        anyhow::bail!(\"startup timeout; last log lines:\\n{}\", log.lines().rev().take(30).collect::<Vec<_>>().join(\"\\n\"));\n    }\n    other => other,\n}","preventionTips":["Stop envs cleanly (`neon_local env stop`) so no process holds ports across runs.","Scale start_timeout/retry_timeout with data size on slow CI disks.","Make status-check closures resilient: Ok(false) for not-ready instead of Err.","Watch {datadir}/{process_name}.log during long startups to distinguish slow from stuck."],"tags":["rust","neon","control-plane","background-process","startup","timeout","health-check"],"backgroundTag":"service-startup-timeout","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}