neondatabase/neon · error · anyhow::Error

safekeeper {sk_id} does not exist

Error message

safekeeper {sk_id} does not exist

What it means

Endpoint::build_safekeepers_connstrs maps the safekeeper NodeIds supplied to an endpoint start (or reconfigure) to connection strings by looking each id up in the LocalEnv's safekeepers list. An id with no matching node in the env raises this error; it is a pure config-coherence failure, not a runtime/network one. Only Primary-mode endpoints resolve safekeeper ids, so read-replica starts with unknown ids pass through silently.

Source

Thrown at control_plane/src/endpoint.rs:680

            Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok("".to_string()),
            Err(e) => Err(anyhow::Error::new(e).context(format!(
                "failed to read config file in {}",
                postgresql_conf_path.to_str().unwrap()
            ))),
        }
    }

    /// Map safekeepers ids to the actual connection strings.
    fn build_safekeepers_connstrs(&self, sk_ids: Vec<NodeId>) -> Result<Vec<String>> {
        let mut safekeeper_connstrings = Vec::new();
        if self.mode == ComputeMode::Primary {
            for sk_id in sk_ids {
                let sk = self
                    .env
                    .safekeepers
                    .iter()
                    .find(|node| node.id == sk_id)
                    .ok_or_else(|| anyhow!("safekeeper {sk_id} does not exist"))?;
                safekeeper_connstrings.push(format!("127.0.0.1:{}", sk.get_compute_port()));
            }
        }
        Ok(safekeeper_connstrings)
    }

    /// Generate a JWT with the correct claims.
    pub fn generate_jwt(&self, scope: Option<ComputeClaimsScope>) -> Result<String> {
        self.env.generate_auth_token(&ComputeClaims {
            audience: match scope {
                Some(ComputeClaimsScope::Admin) => Some(vec![COMPUTE_AUDIENCE.to_owned()]),
                _ => None,
            },
            compute_id: match scope {
                Some(ComputeClaimsScope::Admin) => None,
                _ => Some(self.endpoint_id.clone()),
            },
            scope,

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Check what exists: `neon_local safekeeper list` and compare the ids you pass to endpoint start.
2. Create the missing safekeeper (`neon_local safekeeper create --id <sk_id>`) before starting the endpoint.
3. Drop the unknown id from the endpoint's safekeeper list if it was stale.
4. If ids come from generated config, regenerate it after the safekeeper topology changes.

Example fix

// before
let connstrs = endpoint.build_safekeepers_connstrs(vec![NodeId(3)])?; // safekeeper 3 does not exist

// after
let valid: Vec<NodeId> = requested_ids.iter()
    .copied()
    .filter(|id| env.safekeepers.iter().any(|sk| sk.id == *id))
    .collect();
anyhow::ensure!(valid.len() == requested_ids.len(),
    "unknown safekeeper ids: {:?}",
    requested_ids.iter().filter(|id| !valid.contains(id)).collect::<Vec<_>>());
let connstrs = endpoint.build_safekeepers_connstrs(valid)?;

// resolve ids against the env before starting/reconfiguring the endpoint
let unknown: Vec<_> = sk_ids.iter()
    .filter(|id| !env.safekeepers.iter().any(|sk| sk.id == **id))
    .collect();
anyhow::ensure!(unknown.is_empty(), "unknown safekeeper ids: {unknown:?}");

fn all_safekeepers_exist(env: &LocalEnv, ids: &[NodeId]) -> bool {
    ids.iter().all(|id| env.safekeepers.iter().any(|sk| sk.id == *id))
}

match endpoint.start(args).await {
    Err(e) if e.to_string().contains("safekeeper") && e.to_string().contains("does not exist") => {
        // create missing safekeepers, then retry start
        for id in missing_ids { create_safekeeper(env, id)?; }
        endpoint.start(args).await
    }
    other => other,
}

When it happens

Trigger: Calling endpoint start with --safekeepers 3 when the env only has safekeepers 1 and 2; passing ids parsed from a spec/config file that were never created via `neon_local safekeeper create`; renaming/regenerating an env so previously valid ids no longer exist; reconfiguring an endpoint to a safekeeper set from a different env.

Common situations: Hand-editing scripts after adding/removing safekeepers; copy/pasting start commands between envs; automated harnesses deriving sk ids dynamically while safekeeper creation failed earlier in the script (so the list is shorter than expected).

Related errors

• Endpoint::from_dir_entry failed: '{}' is not a directory
• pg_ctl failed, exit code: {}, stdout: {}, stderr: {}
• `datadir` must be a directory when calling this function: {d
• {} did not start+pass status checks within {:?} seconds
• Failed to send signal to {process_name} with pid {pid}: {e}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/f1d11fd3ca5aafdf. Report an issue: GitHub.