{"record":{"id":"d42f4e4bd634e23e","repo":"neondatabase/neon","slug":"postgres-sync-safekeepers-exited-with-non-zero-s","errorCode":null,"errorMessage":"postgres --sync-safekeepers exited with non-zero status: {}. stdout: {}","messagePattern":"postgres --sync-safekeepers exited with non-zero status: (.+?)\\. stdout: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"compute_tools/src/compute.rs","lineNumber":1544,"sourceCode":"        // will be collected in a child thread.\n        let stderr = sync_handle\n            .stderr\n            .take()\n            .expect(\"stderr should be captured\");\n        let logs_handle = handle_postgres_logs(stderr);\n\n        let sync_output = sync_handle\n            .wait_with_output()\n            .expect(\"postgres --sync-safekeepers failed\");\n        SYNC_SAFEKEEPERS_PID.store(0, Ordering::SeqCst);\n\n        // Process has exited, so we can join the logs thread.\n        let _ = tokio::runtime::Handle::current()\n            .block_on(logs_handle)\n            .map_err(|e| tracing::error!(\"log task panicked: {:?}\", e));\n\n        if !sync_output.status.success() {\n            anyhow::bail!(\n                \"postgres --sync-safekeepers exited with non-zero status: {}. stdout: {}\",\n                sync_output.status,\n                String::from_utf8(sync_output.stdout)\n                    .expect(\"postgres --sync-safekeepers exited, and stdout is not utf-8\"),\n            );\n        }\n\n        self.state.lock().unwrap().metrics.sync_safekeepers_ms = Utc::now()\n            .signed_duration_since(start_time)\n            .to_std()\n            .unwrap()\n            .as_millis() as u64;\n\n        let lsn = Lsn::from_str(String::from_utf8(sync_output.stdout)?.trim())?;\n\n        Ok(lsn)\n    }\n","sourceCodeStart":1526,"sourceCodeEnd":1562,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/compute_tools/src/compute.rs#L1526-L1562","documentation":"Before starting a primary postgres, compute_ctl runs `postgres --sync-safekeepers` to push local WAL to a quorum of safekeepers and waits for it. The helper exited with a non-zero status; the error includes the exit status and captured stdout, which usually names the real cause (unreachable safekeeper, no quorum, auth failure, bad timeline).","triggerScenarios":"The --sync-safekeepers subprocess fails: neon.safekeepers endpoints unreachable from the compute, WAL proposer cannot achieve quorum (majority of safekeepers down), tenant/timeline unknown on the safekeepers, expired/invalid storage auth token, or corrupted local WAL state.","commonSituations":"Endpoint start racing safekeepers that are still booting; network policy blocking compute->safekeeper traffic; a wrong/comma-mangled neon.safekeepers GUC; auth token rotation leaving the compute with stale credentials; safekeeper cluster scaled down below quorum.","solutions":["Read the included stdout and postgres log for the specific failure (connect timeout vs quorum vs auth)","Verify every address in neon.safekeepers resolves and is reachable (pg-compatible port 5678) from the compute node","Check safekeeper health and quorum: a majority must be alive for sync to succeed","If it is a startup race, simply retry starting the endpoint once safekeepers are up","Validate/refresh the storage auth token the compute uses toward the safekeepers"],"exampleFix":"// before: single attempt, any non-zero exit fails endpoint start\nlet sync_output = sync_handle.wait_with_output().expect(\"postgres --sync-safekeepers failed\");\nif !sync_output.status.success() { anyhow::bail!(...); }\n// after: one bounded retry for transient safekeeper unavailability\nfor attempt in 0..2 {\n    if sync_output.status.success() { break; }\n    if attempt == 1 { anyhow::bail!(\"...: {}\", String::from_utf8_lossy(&sync_output.stdout)); }\n    tokio::time::sleep(Duration::from_secs(5)).await;\n}","handlingStrategy":"retry","validationCode":"// Pre-flight: can we reach every safekeeper before invoking sync?\nfor sk in safekeeper_hosts {\n    if tokio::net::TcpStream::connect(&sk).await.is_err() {\n        warn!(\"safekeeper {sk} unreachable before sync-safekeepers\");\n    }\n}","typeGuard":null,"tryCatchPattern":"// Treat sync failure as retryable during endpoint start (safekeepers may still be booting)\nfor attempt in 1..=3 {\n    match run_sync_safekeepers().await {\n        Ok(()) => break,\n        Err(e) if attempt == 3 => return Err(anyhow!(\"sync-safekeepers failed after retries: {e:#}\")),\n        Err(e) => { warn!(\"sync attempt {attempt} failed: {e}\"); tokio::time::sleep(Duration::from_secs(3)).await; }\n    }\n}","preventionTips":["Order infra so safekeepers are healthy before computes start (readiness gates)","Keep neon.safekeepers accurate: all addresses, resolvable, port-reachable from the compute subnet","Alert on safekeeper quorum loss; sync cannot succeed without a majority","Rotate auth tokens before expiry on both compute and safekeeper sides"],"tags":["rust","compute-ctl","postgres","safekeepers","wal","subprocess","startup"],"backgroundTag":"subprocess-exit-nonzero","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}