{"record":{"id":"ca70c6ddf9e963b9","repo":"neondatabase/neon","slug":"failed-to-check-node-status-e-ca70c6","errorCode":null,"errorMessage":"Failed to check node status: {e}","messagePattern":"Failed to check node status: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"control_plane/src/safekeeper.rs","lineNumber":246,"sourceCode":"            args.extend([\"--auth-token-path\".to_owned(), token_path_str]);\n        }\n\n        args.extend_from_slice(extra_opts);\n\n        let env_variables = Vec::new();\n        background_process::start_process(\n            &format!(\"safekeeper-{id}\"),\n            &datadir,\n            &self.env.safekeeper_bin(),\n            &args,\n            env_variables,\n            background_process::InitialPidFile::Expect(self.pid_file()),\n            retry_timeout,\n            || async {\n                match self.check_status().await {\n                    Ok(()) => Ok(true),\n                    Err(SafekeeperHttpError::Transport(_)) => Ok(false),\n                    Err(e) => Err(anyhow::anyhow!(\"Failed to check node status: {e}\")),\n                }\n            },\n        )\n        .await\n    }\n\n    ///\n    /// Stop the server.\n    ///\n    /// If 'immediate' is true, we use SIGQUIT, killing the process immediately.\n    /// Otherwise we use SIGTERM, triggering a clean shutdown\n    ///\n    /// If the server is not running, returns success\n    ///\n    pub fn stop(&self, immediate: bool) -> anyhow::Result<()> {\n        background_process::stop_process(\n            immediate,\n            &format!(\"safekeeper {}\", self.id),","sourceCodeStart":228,"sourceCodeEnd":264,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/safekeeper.rs#L228-L264","documentation":"The same startup gate as the pageserver's, but for a locally started safekeeper: neon_local polls the safekeeper's HTTP status endpoint until it is ready. Only SafekeeperHttpError::Transport failures are retried; any other error (an HTTP error response, decode failure, etc.) stops the retry loop and fails the start with this message.","triggerScenarios":"The safekeeper's HTTP endpoint returns an error status to the probe (auth enabled without a valid token, unknown route due to version skew between neon_local and the safekeeper binary) or a non-transport API error surfaces during polling.","commonSituations":"Mixing binary versions in a dev checkout (old safekeeper without the status route), JWT auth enabled on safekeepers while neon_local lacks credentials, or a safekeeper crashing during startup.","solutions":["Check <base>/safekeeper-<id>/safekeeper.log for the failure during startup","Verify the safekeeper's HTTP listen port matches the env config","Align auth settings: provide keys/tokens or use an unauthenticated local config","Rebuild/update the safekeeper binary so its HTTP API matches what neon_local expects"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before start: verify the configured safekeeper HTTP port is what you expect\nlet addr = &sk_conf.http_listen_addr;\nanyhow::ensure!(addr.port() != 0, \"safekeeper http port must be fixed for status polling\");","typeGuard":null,"tryCatchPattern":"match sk.check_status().await {\n    Ok(()) => {}\n    Err(SafekeeperHttpError::Transport(_)) => { /* retry until timeout */ }\n    Err(e) => { /* HTTP-level error: check auth config and safekeeper-<id>.log */ }\n}","preventionTips":["Keep neon_local and safekeeper binaries from the same build","Align safekeeper auth config with the keys neon_local knows about","Check safekeeper.log whenever the retry loop aborts — transport errors alone are retried"],"tags":["rust","neon-local","safekeeper","http","startup","auth"],"backgroundTag":"service-health-check-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}