neondatabase/neon · error · anyhow::Error

Failed to check node status: {e}

Error message

Failed to check node status: {e}

What it means

While waiting for a locally started pageserver, neon_local polls GET /v1/status on its management HTTP API. Only mgmt_api::Error::ReceiveBody is treated as not-yet-ready and retried until retry_timeout; every other failure — SendRequest (request could not be sent/connect failed), ApiError (an HTTP error status such as 401 or 503), or a malformed error body — aborts the start immediately with this message.

Source

Thrown at control_plane/src/pageserver.rs:336

                self.conf.id, datadir,
            )
        })?;
        let args = vec!["-D", datadir_path_str];

        background_process::start_process(
            "pageserver",
            &datadir,
            &self.env.pageserver_bin(),
            args,
            self.pageserver_env_variables()?,
            background_process::InitialPidFile::Expect(self.pid_file()),
            retry_timeout,
            || async {
                let st = self.check_status().await;
                match st {
                    Ok(()) => Ok(true),
                    Err(mgmt_api::Error::ReceiveBody(_)) => Ok(false),
                    Err(e) => Err(anyhow::anyhow!("Failed to check node status: {e}")),
                }
            },
        )
        .await?;

        Ok(())
    }

    fn pageserver_env_variables(&self) -> anyhow::Result<Vec<(String, String)>> {
        // FIXME: why is this tied to pageserver's auth type? Whether or not the safekeeper
        // needs a token, and how to generate that token, seems independent to whether
        // the pageserver requires a token in incoming requests.
        Ok(if self.conf.http_auth_type != AuthType::Trust {
            // Generate a token to connect from the pageserver to a safekeeper
            let token = self
                .env
                .generate_auth_token(&Claims::new(None, Scope::SafekeeperData))?;
            vec![("NEON_AUTH_TOKEN".to_owned(), token)]

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Check <base>/pageserver_<id>/pageserver.log and the neon_local output for the underlying mgmt_api error (status code and body)
2. Verify listen_http_addr is the port being polled and that nothing else binds it
3. If JWT auth is enabled, ensure neon_local generated/loaded the matching private key; or use http_auth_type = trust for local runs
4. Retry the start once transient causes (slow init, port races) are ruled out

// before start: catch the common fatal cause early
if ps_conf.http_auth_type != AuthType::Trust {
    anyhow::ensure!(
        env.get_private_key_path().exists(),
        "JWT auth enabled but the env has no private key"
    );
}

match ps.check_status().await {
    Ok(()) => {}
    Err(mgmt_api::Error::ReceiveBody(_)) => { /* still starting: keep polling within retry_timeout */ }
    Err(mgmt_api::Error::ApiError(status, body)) => {
        // 401/403 => fix auth; 503 => keep waiting; inspect `body`
    }
    Err(other) => { /* send/connect problem: check port, process, pageserver.log */ }
}

When it happens

Trigger: The pageserver's HTTP endpoint answers the status probe with an error status (401 when JWT auth is configured but neon_local sends no/wrong token, 503 while internal services initialize) or the management API request fails at the send/connect level in a non-retryable way.

Common situations: Auth-type mismatch between the neon config and neon_local's token generation, a port collision making another service answer on listen_http_addr, or a pageserver that crashes mid-startup.

Related errors

• Failed to check node status: {e}
• pageserver connection information should be provided
• shard {shard_index} missing from pageserver_connection_info
• expected a directory, got {:?}
• id mismatch: identity.toml:id={identity_toml_id} pageserver_

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/4f2faa383187bd26. Report an issue: GitHub.