neondatabase/neon · error

process failed to start: {e}

Error message

process failed to start: {e}

What it means

During startup polling, process_started() runs the caller-supplied status_check future; the future itself returning Err (as opposed to Ok(false)) is immediately wrapped as `process failed to start: {e}`. That is, the health-check machinery errored (network failure, bad URL, deserialization), not merely reported 'not ready'. start_process prints the chained error and aborts the retry loop when it sees this.

Source

Thrown at control_plane/src/background_process.rs:385

}

async fn process_started<F, Fut>(
    pid: Pid,
    pid_file_to_check: &Utf8Path,
    status_check: &F,
) -> anyhow::Result<bool>
where
    F: Fn() -> Fut,
    Fut: std::future::Future<Output = anyhow::Result<bool>>,
{
    match status_check().await {
        Ok(true) => match pid_file::read(pid_file_to_check)? {
            PidFileRead::NotExist => Ok(false),
            PidFileRead::LockedByOtherProcess(pid_in_file) => Ok(pid_in_file == pid),
            PidFileRead::NotHeldByAnyProcess(_) => Ok(false),
        },
        Ok(false) => Ok(false),
        Err(e) => anyhow::bail!("process failed to start: {e}"),
    }
}

pub(crate) fn process_has_stopped(pid: Pid) -> anyhow::Result<bool> {
    match kill(pid, None) {
        // Process exists, keep waiting
        Ok(_) => Ok(false),
        // Process not found, we're done
        Err(Errno::ESRCH) => Ok(true),
        Err(err) => anyhow::bail!("Failed to send signal to process with pid {pid}: {err}"),
    }
}

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Look at the {e:#} chain printed with the error — it names the real failure (connection refused, decode error, etc.).
2. Confirm the process binary actually stays alive: check {datadir}/{process_name}.log and `ps` for the pid.
3. Curl the exact URL the status check uses, from the same host, and compare ports with what the process logs as its listen address.
4. If the error is a serde/decode error, align control_plane and process versions (same git checkout / cargo workspace) so /status schemas match.
5. Make the status_check closure distinguish 'not ready' (Ok(false)) from hard errors so transient connection-refused during early startup doesn't abort the loop.

Example fix

// before
let check = || async { client.get(&url).send().await?.json::<Status>().await.map(|_| true) };
// any transport/decode error aborts startup as "process failed to start"

// after
let check = || async {
    match client.get(&url).send().await {
        Ok(resp) if resp.status().is_success() => Ok(resp.json::<Status>().await.map(|_| true)?),
        Ok(_) => Ok(false),          // not ready yet, keep polling
        Err(e) if e.is_connect() => Ok(false), // process still booting
        Err(e) => Err(e.into()),     // real failure
    }
};

// make the status check non-erroring for transient states before passing it in
let check = || async {
    match client.get(&url).send().await {
        Ok(r) if r.status().is_success() => Ok(r.json::<Status>().await.map(|_| true).unwrap_or(false)),
        Ok(_) => Ok(false),
        Err(e) if e.is_connect() || e.is_timeout() => Ok(false),
        Err(_) => Ok(false),
    }
};

match start_process(...).await {
    Err(e) if e.to_string().contains("process failed to start") => {
        // inspect {e:#} chain; fix root cause (crashed binary / bad check URL) then retry
        anyhow::bail!("startup aborted by status-check error: {e:#}");
    }
    other => other,
}

When it happens

Trigger: Calling start_process with a status check that does an HTTP GET to the process's mgmt/status port: connection refused because the process crashed and the port closed, DNS/address typo in the check URL, response body failing JSON deserialization after an API change, or auth rejection surfacing as an error rather than false.

Common situations: Compute/pageserver binary crashing between spawn and first health poll (so connect fails with ECONNREFUSED); version skew where the /status JSON schema changed and serde fails; the status closure pointing at external_http_address while the process listens only on internal; TLS/auth middleware erroring on self-signed certs in the check client.

Related errors

• {} did not start+pass status checks within {:?} seconds
• `datadir` must be a directory when calling this function: {d
• Failed to send signal to {process_name} with pid {pid}: {e}
• {} with pid {} did not stop in {:?} seconds
• Failed to send signal to process with pid {pid}: {err}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/eddc0790f14833f6. Report an issue: GitHub.