{"record":{"id":"eddc0790f14833f6","repo":"neondatabase/neon","slug":"process-failed-to-start-e","errorCode":null,"errorMessage":"process failed to start: {e}","messagePattern":"process failed to start: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"control_plane/src/background_process.rs","lineNumber":385,"sourceCode":"}\n\nasync fn process_started<F, Fut>(\n    pid: Pid,\n    pid_file_to_check: &Utf8Path,\n    status_check: &F,\n) -> anyhow::Result<bool>\nwhere\n    F: Fn() -> Fut,\n    Fut: std::future::Future<Output = anyhow::Result<bool>>,\n{\n    match status_check().await {\n        Ok(true) => match pid_file::read(pid_file_to_check)? {\n            PidFileRead::NotExist => Ok(false),\n            PidFileRead::LockedByOtherProcess(pid_in_file) => Ok(pid_in_file == pid),\n            PidFileRead::NotHeldByAnyProcess(_) => Ok(false),\n        },\n        Ok(false) => Ok(false),\n        Err(e) => anyhow::bail!(\"process failed to start: {e}\"),\n    }\n}\n\npub(crate) fn process_has_stopped(pid: Pid) -> anyhow::Result<bool> {\n    match kill(pid, None) {\n        // Process exists, keep waiting\n        Ok(_) => Ok(false),\n        // Process not found, we're done\n        Err(Errno::ESRCH) => Ok(true),\n        Err(err) => anyhow::bail!(\"Failed to send signal to process with pid {pid}: {err}\"),\n    }\n}\n","sourceCodeStart":367,"sourceCodeEnd":398,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/background_process.rs#L367-L398","documentation":"During startup polling, process_started() runs the caller-supplied status_check future; the future itself returning Err (as opposed to Ok(false)) is immediately wrapped as `process failed to start: {e}`. That is, the health-check machinery errored (network failure, bad URL, deserialization), not merely reported 'not ready'. start_process prints the chained error and aborts the retry loop when it sees this.","triggerScenarios":"Calling start_process with a status check that does an HTTP GET to the process's mgmt/status port: connection refused because the process crashed and the port closed, DNS/address typo in the check URL, response body failing JSON deserialization after an API change, or auth rejection surfacing as an error rather than false.","commonSituations":"Compute/pageserver binary crashing between spawn and first health poll (so connect fails with ECONNREFUSED); version skew where the /status JSON schema changed and serde fails; the status closure pointing at external_http_address while the process listens only on internal; TLS/auth middleware erroring on self-signed certs in the check client.","solutions":["Look at the {e:#} chain printed with the error — it names the real failure (connection refused, decode error, etc.).","Confirm the process binary actually stays alive: check {datadir}/{process_name}.log and `ps` for the pid.","Curl the exact URL the status check uses, from the same host, and compare ports with what the process logs as its listen address.","If the error is a serde/decode error, align control_plane and process versions (same git checkout / cargo workspace) so /status schemas match.","Make the status_check closure distinguish 'not ready' (Ok(false)) from hard errors so transient connection-refused during early startup doesn't abort the loop."],"exampleFix":"// before\nlet check = || async { client.get(&url).send().await?.json::<Status>().await.map(|_| true) };\n// any transport/decode error aborts startup as \"process failed to start\"\n\n// after\nlet check = || async {\n    match client.get(&url).send().await {\n        Ok(resp) if resp.status().is_success() => Ok(resp.json::<Status>().await.map(|_| true)?),\n        Ok(_) => Ok(false),          // not ready yet, keep polling\n        Err(e) if e.is_connect() => Ok(false), // process still booting\n        Err(e) => Err(e.into()),     // real failure\n    }\n};","handlingStrategy":"retry","validationCode":"// make the status check non-erroring for transient states before passing it in\nlet check = || async {\n    match client.get(&url).send().await {\n        Ok(r) if r.status().is_success() => Ok(r.json::<Status>().await.map(|_| true).unwrap_or(false)),\n        Ok(_) => Ok(false),\n        Err(e) if e.is_connect() || e.is_timeout() => Ok(false),\n        Err(_) => Ok(false),\n    }\n};","typeGuard":null,"tryCatchPattern":"match start_process(...).await {\n    Err(e) if e.to_string().contains(\"process failed to start\") => {\n        // inspect {e:#} chain; fix root cause (crashed binary / bad check URL) then retry\n        anyhow::bail!(\"startup aborted by status-check error: {e:#}\");\n    }\n    other => other,\n}","preventionTips":["Distinguish 'not ready' (Ok(false)) from hard errors in every status-check closure.","Keep control_plane and the supervised process versions in lockstep so status schemas match.","Curl the exact status URL with the same auth during bring-up debugging."],"tags":["rust","neon","control-plane","background-process","health-check","startup"],"backgroundTag":"health-check-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}