neondatabase/neon · error

could not get backends state change

Error message

could not get backends state change: {}

What it means

ComputeMonitor::check calls get_backends_state_change(), which SELECTs state and to_char(state_change, 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') FROM pg_stat_activity for every client backend except the monitor's own pid and the cloud_admin user. This error means that query itself returned Err, so the monitor could not compute the most-recent idle state-change timestamp used in the suspend decision. As with all check() failures, the monitor logs it with downtime info, reports downtime, and reconnects for the next tick.

Solutions

1. Inspect the appended source error (e) for the tokio_postgres cause and SQLSTATE
2. Confirm Postgres is up and the compute is in Running state; the monitor will self-heal by reconnecting next iteration
3. If errors persist every 500ms, check Postgres logs for crashes/restarts and network issues inside the compute container
4. Treat consecutive errors, not single events, as actionable downtime signal

match get_backends_state_change(cli) {
    Ok(last_active) => { /* compare and update self.last_active */ }
    Err(e) => {
        error!("could not get backends state change: {e}");
        client = conf.connect(NoTls); // retry on next tick
    }
}

When it happens

Trigger: cli.query() over pg_stat_activity fails: broken connection, statement timeout, Postgres restarting, or the stats view being temporarily unavailable while the server changes state.

Common situations: Compute suspend/resume transitions racing the 500ms monitor poll; long lock waits on pg_stat_activity under heavy load; monitor thread surviving a Postgres crash by design and logging this each tick until Postgres returns.

Related errors

• could not query backends
• could not get database statistics
• could not query active_time
• failed to get list of autovacuum workers
• connection to postgres closed

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/22f9bf4782465dcc. Report an issue: GitHub.

Appendix: source

Thrown at compute_tools/src/monitor.rs:264

        // can return None or stale timestamp, so it's `compute.update_last_active()`
        // responsibility to check if the new timestamp is more recent than the current one.
        // This helps us to discover new sessions that have not done anything yet.
        match get_backends_state_change(cli) {
            Ok(last_active) => match (last_active, self.last_active) {
                (Some(last_active), Some(prev_last_active)) => {
                    if last_active > prev_last_active {
                        self.last_active = Some(last_active);
                        return Ok(());
                    }
                }
                (Some(last_active), None) => {
                    self.last_active = Some(last_active);
                    return Ok(());
                }
                _ => {}
            },
            Err(e) => {
                return Err(anyhow::anyhow!(
                    "could not get backends state change: {}",
                    e
                ));
            }
        }

        // If there are existing (logical) walsenders, do not suspend.
        //
        // N.B. walproposer doesn't currently show up in pg_stat_replication,
        // but protect if it will.
        const WS_COUNT_QUERY: &str =
            "select count(*) from pg_stat_replication where application_name != 'walproposer';";
        match cli.query_one(WS_COUNT_QUERY, &[]) {
            Ok(r) => match r.try_get::<&str, i64>("count") {
                Ok(num_ws) => {
                    if num_ws > 0 {
                        self.last_active = Some(Utc::now());
                        return Ok(());

View on GitHub (pinned to 8f60b04da4)