What causes "could not query backends: {}" in neondatabase/neon?

Read the nested error for cause and SQLSTATE

could not query backends: {}

Error message

could not query backends: {}

What it means

get_backends_state_change() SELECTs state and a formatted state_change timestamp FROM pg_stat_activity for client backends (excluding the monitor's own pid and cloud_admin). This error means cli.query() itself returned Err, so the idle-timestamp computation could not run. It flows out of check() where the monitor logs it, counts downtime, and reconnects.

Source

Thrown at compute_tools/src/monitor.rs:492

                        Err(e) => {
                            info!("cannot parse backend state_change DateTime: {}", e);
                            continue;
                        }
                    }
                } else {
                    // Found non-idle backend, so the last activity is NOW.
                    // Return immediately, no need to check other backends.
                    return Ok(Some(Utc::now()));
                }
            }

            // Get idle backend `state_change` with the max timestamp.
            if let Some(last) = idle_backs.iter().max() {
                last_active = Some(*last);
            }
        }
        Err(e) => {
            return Err(anyhow::anyhow!("could not query backends: {}", e));
        }
    }

    Ok(last_active)
}

/// Launch a separate compute monitor thread and return its `JoinHandle`.
pub fn launch_monitor(compute: &Arc<ComputeNode>) -> thread::JoinHandle<()> {
    let compute = Arc::clone(compute);
    let experimental = compute.has_feature(ComputeFeature::ActivityMonitorExperimental);
    let now = Utc::now();
    let mut monitor = ComputeMonitor {
        compute,
        last_active: None,
        last_checked: now,
        last_up: now,
        active_time: None,
        sessions: None,

View on GitHub (pinned to 8f60b04da4)

Solutions

Read the nested error for cause and SQLSTATE
Rely on monitor self-healing for one-off failures; it reconnects each 500ms tick
If continuous, examine Postgres logs and connectivity between compute_ctl and the server
Alert on consecutive monitor errors rather than individual occurrences

Defensive patterns

Strategy: retry

Try / catch

let backends = match cli.query(BACKENDS_SQL, &[]) {
    Ok(rows) => rows,
    Err(e) => {
        error!("could not query backends: {e}");
        client = conf.connect(NoTls);
        continue;
    }
};

Prevention

Retry transient pg_stat_activity failures on the next monitor tick
Alert on consecutive errors, not individual ones
Verify Postgres health when the error repeats rapidly

When it happens

Trigger: The pg_stat_activity query fails: connection closed mid-poll, statement timeout, Postgres restarting, or server error while iterating backends.

Common situations: Monitor polling during suspend/resume or crash/restart of Postgres; timeouts under heavy backend churn; transient transport breakage inside the compute container.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/781fe44a2e2dc872. Report an issue: GitHub.