What causes "Failed to check node status: {e}" in neondatabase/neon?

Check /safekeeper- /safekeeper.log for the failure during startup

neondatabase/neon · error · anyhow::Error

Failed to check node status: {e}

Error message

Failed to check node status: {e}

What it means

The same startup gate as the pageserver's, but for a locally started safekeeper: neon_local polls the safekeeper's HTTP status endpoint until it is ready. Only SafekeeperHttpError::Transport failures are retried; any other error (an HTTP error response, decode failure, etc.) stops the retry loop and fails the start with this message.

Source

Thrown at control_plane/src/safekeeper.rs:246

            args.extend(["--auth-token-path".to_owned(), token_path_str]);
        }

        args.extend_from_slice(extra_opts);

        let env_variables = Vec::new();
        background_process::start_process(
            &format!("safekeeper-{id}"),
            &datadir,
            &self.env.safekeeper_bin(),
            &args,
            env_variables,
            background_process::InitialPidFile::Expect(self.pid_file()),
            retry_timeout,
            || async {
                match self.check_status().await {
                    Ok(()) => Ok(true),
                    Err(SafekeeperHttpError::Transport(_)) => Ok(false),
                    Err(e) => Err(anyhow::anyhow!("Failed to check node status: {e}")),
                }
            },
        )
        .await
    }

    ///
    /// Stop the server.
    ///
    /// If 'immediate' is true, we use SIGQUIT, killing the process immediately.
    /// Otherwise we use SIGTERM, triggering a clean shutdown
    ///
    /// If the server is not running, returns success
    ///
    pub fn stop(&self, immediate: bool) -> anyhow::Result<()> {
        background_process::stop_process(
            immediate,
            &format!("safekeeper {}", self.id),

View on GitHub (pinned to 8f60b04da4)

Solutions

Check <base>/safekeeper-<id>/safekeeper.log for the failure during startup
Verify the safekeeper's HTTP listen port matches the env config
Align auth settings: provide keys/tokens or use an unauthenticated local config
Rebuild/update the safekeeper binary so its HTTP API matches what neon_local expects

Defensive patterns

Strategy: retry

Validate before calling

// before start: verify the configured safekeeper HTTP port is what you expect
let addr = &sk_conf.http_listen_addr;
anyhow::ensure!(addr.port() != 0, "safekeeper http port must be fixed for status polling");

Try / catch

match sk.check_status().await {
    Ok(()) => {}
    Err(SafekeeperHttpError::Transport(_)) => { /* retry until timeout */ }
    Err(e) => { /* HTTP-level error: check auth config and safekeeper-<id>.log */ }
}

Prevention

Keep neon_local and safekeeper binaries from the same build
Align safekeeper auth config with the keys neon_local knows about
Check safekeeper.log whenever the retry loop aborts — transport errors alone are retried

When it happens

Trigger: The safekeeper's HTTP endpoint returns an error status to the probe (auth enabled without a valid token, unknown route due to version skew between neon_local and the safekeeper binary) or a non-transport API error surfaces during polling.

Common situations: Mixing binary versions in a dev checkout (old safekeeper without the status route), JWT auth enabled on safekeepers while neon_local lacks credentials, or a safekeeper crashing during startup.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/ca70c6ddf9e963b9. Report an issue: GitHub.