What causes "could not query backends: {}" in neondatabase/neon?

Read the nested error for cause and SQLSTATE

could not query backends

Error message

could not query backends: {}

What it means

get_backends_state_change() SELECTs state and a formatted state_change timestamp FROM pg_stat_activity for client backends (excluding the monitor's own pid and cloud_admin). This error means cli.query() itself returned Err, so the idle-timestamp computation could not run. It flows out of check() where the monitor logs it, counts downtime, and reconnects.

Solutions

Read the nested error for cause and SQLSTATE
Rely on monitor self-healing for one-off failures; it reconnects each 500ms tick
If continuous, examine Postgres logs and connectivity between compute_ctl and the server
Alert on consecutive monitor errors rather than individual occurrences

Defensive patterns

Strategy: retry

Try / catch

let backends = match cli.query(BACKENDS_SQL, &[]) {
    Ok(rows) => rows,
    Err(e) => {
        error!("could not query backends: {e}");
        client = conf.connect(NoTls);
        continue;
    }
};

Prevention

Retry transient pg_stat_activity failures on the next monitor tick
Alert on consecutive errors, not individual ones
Verify Postgres health when the error repeats rapidly

When it happens

Trigger: The pg_stat_activity query fails: connection closed mid-poll, statement timeout, Postgres restarting, or server error while iterating backends.

Common situations: Monitor polling during suspend/resume or crash/restart of Postgres; timeouts under heavy backend churn; transient transport breakage inside the compute container.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/781fe44a2e2dc872. Report an issue: GitHub.

Appendix: source

Thrown at compute_tools/src/monitor.rs:492

                        Err(e) => {
                            info!("cannot parse backend state_change DateTime: {}", e);
                            continue;
                        }
                    }
                } else {
                    // Found non-idle backend, so the last activity is NOW.
                    // Return immediately, no need to check other backends.
                    return Ok(Some(Utc::now()));
                }
            }

            // Get idle backend `state_change` with the max timestamp.
            if let Some(last) = idle_backs.iter().max() {
                last_active = Some(*last);
            }
        }
        Err(e) => {
            return Err(anyhow::anyhow!("could not query backends: {}", e));
        }
    }

    Ok(last_active)
}

/// Launch a separate compute monitor thread and return its `JoinHandle`.
pub fn launch_monitor(compute: &Arc<ComputeNode>) -> thread::JoinHandle<()> {
    let compute = Arc::clone(compute);
    let experimental = compute.has_feature(ComputeFeature::ActivityMonitorExperimental);
    let now = Utc::now();
    let mut monitor = ComputeMonitor {
        compute,
        last_active: None,
        last_checked: now,
        last_up: now,
        active_time: None,
        sessions: None,

View on GitHub (pinned to 8f60b04da4)