postgres --sync-safekeepers exited with non-zero status: {}.

Error message

postgres --sync-safekeepers exited with non-zero status: {}. stdout: {}

What it means

Before starting a primary postgres, compute_ctl runs `postgres --sync-safekeepers` to push local WAL to a quorum of safekeepers and waits for it. The helper exited with a non-zero status; the error includes the exit status and captured stdout, which usually names the real cause (unreachable safekeeper, no quorum, auth failure, bad timeline).

Source

Thrown at compute_tools/src/compute.rs:1544

        // will be collected in a child thread.
        let stderr = sync_handle
            .stderr
            .take()
            .expect("stderr should be captured");
        let logs_handle = handle_postgres_logs(stderr);

        let sync_output = sync_handle
            .wait_with_output()
            .expect("postgres --sync-safekeepers failed");
        SYNC_SAFEKEEPERS_PID.store(0, Ordering::SeqCst);

        // Process has exited, so we can join the logs thread.
        let _ = tokio::runtime::Handle::current()
            .block_on(logs_handle)
            .map_err(|e| tracing::error!("log task panicked: {:?}", e));

        if !sync_output.status.success() {
            anyhow::bail!(
                "postgres --sync-safekeepers exited with non-zero status: {}. stdout: {}",
                sync_output.status,
                String::from_utf8(sync_output.stdout)
                    .expect("postgres --sync-safekeepers exited, and stdout is not utf-8"),
            );
        }

        self.state.lock().unwrap().metrics.sync_safekeepers_ms = Utc::now()
            .signed_duration_since(start_time)
            .to_std()
            .unwrap()
            .as_millis() as u64;

        let lsn = Lsn::from_str(String::from_utf8(sync_output.stdout)?.trim())?;

        Ok(lsn)
    }

View on GitHub (pinned to 8f60b04da4)

Solutions

Read the included stdout and postgres log for the specific failure (connect timeout vs quorum vs auth)
Verify every address in neon.safekeepers resolves and is reachable (pg-compatible port 5678) from the compute node
Check safekeeper health and quorum: a majority must be alive for sync to succeed
If it is a startup race, simply retry starting the endpoint once safekeepers are up
Validate/refresh the storage auth token the compute uses toward the safekeepers

Example fix

// before: single attempt, any non-zero exit fails endpoint start
let sync_output = sync_handle.wait_with_output().expect("postgres --sync-safekeepers failed");
if !sync_output.status.success() { anyhow::bail!(...); }
// after: one bounded retry for transient safekeeper unavailability
for attempt in 0..2 {
    if sync_output.status.success() { break; }
    if attempt == 1 { anyhow::bail!("...: {}", String::from_utf8_lossy(&sync_output.stdout)); }
    tokio::time::sleep(Duration::from_secs(5)).await;
}

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: can we reach every safekeeper before invoking sync?
for sk in safekeeper_hosts {
    if tokio::net::TcpStream::connect(&sk).await.is_err() {
        warn!("safekeeper {sk} unreachable before sync-safekeepers");
    }
}

Try / catch

// Treat sync failure as retryable during endpoint start (safekeepers may still be booting)
for attempt in 1..=3 {
    match run_sync_safekeepers().await {
        Ok(()) => break,
        Err(e) if attempt == 3 => return Err(anyhow!("sync-safekeepers failed after retries: {e:#}")),
        Err(e) => { warn!("sync attempt {attempt} failed: {e}"); tokio::time::sleep(Duration::from_secs(3)).await; }
    }
}

Prevention

Order infra so safekeepers are healthy before computes start (readiness gates)
Keep neon.safekeepers accurate: all addresses, resolvable, port-reachable from the compute subnet
Alert on safekeeper quorum loss; sync cannot succeed without a majority
Rotate auth tokens before expiry on both compute and safekeeper sides

When it happens

Trigger: The --sync-safekeepers subprocess fails: neon.safekeepers endpoints unreachable from the compute, WAL proposer cannot achieve quorum (majority of safekeepers down), tenant/timeline unknown on the safekeepers, expired/invalid storage auth token, or corrupted local WAL state.

Common situations: Endpoint start racing safekeepers that are still booting; network policy blocking compute->safekeeper traffic; a wrong/comma-mangled neon.safekeepers GUC; auth token rotation leaving the compute with stale credentials; safekeeper cluster scaled down below quorum.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/d42f4e4bd634e23e. Report an issue: GitHub.