{"record":{"id":"e6a7735011159553","repo":"neondatabase/neon","slug":"connection-to-postgres-closed","errorCode":null,"errorMessage":"connection to postgres closed","messagePattern":"connection to postgres closed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"compute_tools/src/checker.rs","lineNumber":15,"sourceCode":"use anyhow::{Ok, Result, anyhow};\nuse tokio_postgres::NoTls;\nuse tracing::{error, instrument, warn};\n\nuse crate::compute::ComputeNode;\n\n/// Update timestamp in a row in a special service table to check\n/// that we can actually write some data in this particular timeline.\n#[instrument(skip_all)]\npub async fn check_writability(compute: &ComputeNode) -> Result<()> {\n    // Connect to the database.\n    let conf = compute.get_tokio_conn_conf(Some(\"compute_ctl:availability_checker\"));\n    let (client, connection) = conf.connect(NoTls).await?;\n    if client.is_closed() {\n        return Err(anyhow!(\"connection to postgres closed\"));\n    }\n\n    // The connection object performs the actual communication with the database,\n    // so spawn it off to run on its own.\n    tokio::spawn(async move {\n        if let Err(e) = connection.await {\n            error!(\"connection error: {}\", e);\n        }\n    });\n\n    let query = \"\n    INSERT INTO public.health_check VALUES (1, pg_catalog.now())\n        ON CONFLICT (id) DO UPDATE\n         SET updated_at = pg_catalog.now();\";\n\n    match client.simple_query(query).await {\n        Result::Ok(result) => {\n            if result.len() != 1 {","sourceCodeStart":1,"sourceCodeEnd":33,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/compute_tools/src/checker.rs#L1-L33","documentation":"The compute_ctl availability checker connected to PostgreSQL via tokio-postgres and then found client.is_closed() == true, meaning the connection task already terminated before any query ran. The checker refuses to run the health-check INSERT because the socket is dead. It indicates the server closed the session between startup and first use rather than a connect-time refusal.","triggerScenarios":"check_writability() runs, conf.connect(NoTls).await succeeds, but the backend terminates immediately: postgres is crashing or shutting down, the session is killed on startup (out of connections, out of memory, terminator), or the server closes the socket during startup packet exchange.","commonSituations":"PostgreSQL OOM-killed or restarting due to a bad GUC while compute_ctl polls availability; max_connections exhausted so the new backend is terminated right after fork; a compute VM under memory pressure; postgres still in recovery/crash-loop during endpoint start.","solutions":["Inspect postgres logs under <pgdata>/log for the termination reason (crash, OOM, admin command)","Verify the postgres process is alive and accepting connections (pg_isready) and that max_connections is not exhausted","Retry the availability check after postgres stabilizes; compute_ctl re-runs the checker periodically","Check pod/VM memory limits and raise them if postgres is being killed"],"exampleFix":"// before\nlet (client, connection) = conf.connect(NoTls).await?;\nif client.is_closed() {\n    return Err(anyhow!(\"connection to postgres closed\"));\n}\n// after: tolerate a just-closed session by retrying the whole connect\nlet (client, connection) = conf.connect(NoTls).await?;\nif client.is_closed() {\n    tracing::warn!(\"postgres connection closed immediately; retrying\");\n    return Err(anyhow!(\"connection to postgres closed\")); // keep error, but surface in retry loop upstream\n}","handlingStrategy":"retry","validationCode":"// Cheap pre-checks before running the writability probe\nif !pgdata_path.join(\"postmaster.pid\").exists() { /* postgres not running */ }\nlet (client, conn) = conf.connect(NoTls).await?;\nif client.is_closed() { /* retry connect instead of proceeding */ }","typeGuard":null,"tryCatchPattern":"// In the checker loop: treat closed-connection as transient\nmatch client.simple_query(query).await {\n    Ok(_) => Ok(()),\n    Err(e) if e.is_closed() => { tracing::warn!(\"pg connection lost, will retry\"); Ok(()) } // checker re-runs\n    Err(e) => Err(anyhow!(\"writability check failed: {e}\")),\n}","preventionTips":["Supervise postgres and let the checker retry on failure rather than treating one failure as fatal","Keep postgres logs (log_directory under pgdata) shipped so the close reason is visible","Set max_connections and memory limits with headroom so new sessions are not terminated at startup"],"tags":["rust","postgres","tokio-postgres","connection","health-check","compute-ctl"],"backgroundTag":"database-connection-closed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}