{"record":{"id":"22f9bf4782465dcc","repo":"neondatabase/neon","slug":"could-not-get-backends-state-change","errorCode":null,"errorMessage":"could not get backends state change: {}","messagePattern":"could not get backends state change: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"compute_tools/src/monitor.rs","lineNumber":264,"sourceCode":"        // can return None or stale timestamp, so it's `compute.update_last_active()`\n        // responsibility to check if the new timestamp is more recent than the current one.\n        // This helps us to discover new sessions that have not done anything yet.\n        match get_backends_state_change(cli) {\n            Ok(last_active) => match (last_active, self.last_active) {\n                (Some(last_active), Some(prev_last_active)) => {\n                    if last_active > prev_last_active {\n                        self.last_active = Some(last_active);\n                        return Ok(());\n                    }\n                }\n                (Some(last_active), None) => {\n                    self.last_active = Some(last_active);\n                    return Ok(());\n                }\n                _ => {}\n            },\n            Err(e) => {\n                return Err(anyhow::anyhow!(\n                    \"could not get backends state change: {}\",\n                    e\n                ));\n            }\n        }\n\n        // If there are existing (logical) walsenders, do not suspend.\n        //\n        // N.B. walproposer doesn't currently show up in pg_stat_replication,\n        // but protect if it will.\n        const WS_COUNT_QUERY: &str =\n            \"select count(*) from pg_stat_replication where application_name != 'walproposer';\";\n        match cli.query_one(WS_COUNT_QUERY, &[]) {\n            Ok(r) => match r.try_get::<&str, i64>(\"count\") {\n                Ok(num_ws) => {\n                    if num_ws > 0 {\n                        self.last_active = Some(Utc::now());\n                        return Ok(());","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/compute_tools/src/monitor.rs#L246-L282","documentation":"ComputeMonitor::check calls get_backends_state_change(), which SELECTs state and to_char(state_change, 'YYYY-MM-DD\"T\"HH24:MI:SS.US\"Z\"') FROM pg_stat_activity for every client backend except the monitor's own pid and the cloud_admin user. This error means that query itself returned Err, so the monitor could not compute the most-recent idle state-change timestamp used in the suspend decision. As with all check() failures, the monitor logs it with downtime info, reports downtime, and reconnects for the next tick.","triggerScenarios":"cli.query() over pg_stat_activity fails: broken connection, statement timeout, Postgres restarting, or the stats view being temporarily unavailable while the server changes state.","commonSituations":"Compute suspend/resume transitions racing the 500ms monitor poll; long lock waits on pg_stat_activity under heavy load; monitor thread surviving a Postgres crash by design and logging this each tick until Postgres returns.","solutions":["Inspect the appended source error (e) for the tokio_postgres cause and SQLSTATE","Confirm Postgres is up and the compute is in Running state; the monitor will self-heal by reconnecting next iteration","If errors persist every 500ms, check Postgres logs for crashes/restarts and network issues inside the compute container","Treat consecutive errors, not single events, as actionable downtime signal"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match get_backends_state_change(cli) {\n    Ok(last_active) => { /* compare and update self.last_active */ }\n    Err(e) => {\n        error!(\"could not get backends state change: {e}\");\n        client = conf.connect(NoTls); // retry on next tick\n    }\n}","preventionTips":["Expect single failures around suspend/resume and restarts; monitor error streaks instead","Keep cloud_admin grants on pg_stat_activity intact","Watch the PG_CURR_DOWNTIME_MS metric rather than parsing individual log lines"],"tags":["postgres","pg-stat-activity","monitoring","rust","connection"],"backgroundTag":"postgres-query-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}