Exhausted all attempts to retrieve the config from the contr

Error message

Exhausted all attempts to retrieve the config from the control plane

What it means

get_config_from_control_plane made 3 sequential GET requests (100ms apart) to {base_uri}/compute/api/v2/computes/{compute_id}/spec, authenticated with the NEON_CONTROL_PLANE_TOKEN env var, and every attempt failed with a retryable error: a reqwest send error (DNS failure, connection refused, TLS problem), HTTP 503, or HTTP 502 (a known intermittent gateway issue). Non-retryable statuses such as 404/500 bail out immediately with a different message, so this error specifically indicates connectivity or availability trouble. At initial startup it fails the run; the configurator's periodic loop keeps retrying afterwards.

Source

Thrown at compute_tools/src/spec.rs:132

                    Err(anyhow!(msg))
                } else {
                    bail!(msg);
                }
            }
        };

        if let Err(e) = &result {
            error!("attempt {} to get config failed with: {}", attempt, e);
        } else {
            return result;
        }

        attempt += 1;
        std::thread::sleep(std::time::Duration::from_millis(100));
    }

    // All attempts failed, return error.
    Err(anyhow::anyhow!(
        "Exhausted all attempts to retrieve the config from the control plane"
    ))
}

/// Check `pg_hba.conf` and update if needed to allow external connections.
pub fn update_pg_hba(pgdata_path: &Path, databricks_pg_hba: Option<&String>) -> Result<()> {
    // XXX: consider making it a part of config.json
    let pghba_path = pgdata_path.join("pg_hba.conf");

    // Update pg_hba to contains databricks specfic settings before adding neon settings
    // PG uses the first record that matches to perform authentication, so we need to have
    // our rules before the default ones from neon.
    // See https://www.postgresql.org/docs/current/auth-pg-hba-conf.html
    if let Some(databricks_pg_hba) = databricks_pg_hba {
        if config::line_in_file(
            &pghba_path,
            &format!("include_if_exists {}\n", *databricks_pg_hba),
        )? {

View on GitHub (pinned to 8f60b04da4)

Solutions

Read the preceding 'attempt N to get config failed with: ...' log lines - they contain the actual reqwest error or HTTP status
Verify reachability: curl -H "Authorization: Bearer $NEON_CONTROL_PLANE_TOKEN" $URI/compute/api/v2/computes/$COMPUTE_ID/spec
Fix the control-plane URI, DNS, or network policy so the request can succeed
For transient 502/503 outages no action is needed - the configurator loop re-requests periodically and recovers automatically

Defensive patterns

Strategy: retry

Validate before calling

# before booting, prove the spec endpoint answers
curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $NEON_CONTROL_PLANE_TOKEN" \
  "$CONTROL_PLANE_URI/compute/api/v2/computes/$COMPUTE_ID/spec"

Try / catch

// caller-side retry with backoff on top of the built-in 3 attempts
let mut backoff = Duration::from_millis(500);
loop {
    match get_config_from_control_plane(uri, &compute_id) {
        Ok(cfg) => break cfg,
        Err(e) if retries_left() => {
            warn!("config fetch failed: {e}; retrying in {:?}", backoff);
            thread::sleep(backoff);
            backoff *= 2;
        }
        Err(e) => return Err(e),
    }
}

Prevention

Validate control-plane URI, DNS and NEON_CONTROL_PLANE_TOKEN as part of environment checks at startup
Retry with backoff at the caller during control-plane deploys - the configurator loop already does this
Watch the 'attempt N to get config failed' lines to distinguish 502/503 (wait it out) from DNS/egress errors (fix config)

When it happens

Trigger: All three do_control_plane_request attempts return retryable failures: control-plane DNS/service unreachable, connection refused, TLS errors, 502 from the gateway, or 503 during maintenance.

Common situations: Dev environments with a wrong --control-plane-uri; Kubernetes NetworkPolicy blocking egress; control-plane redeploy or outage in progress; the compute booting faster than the control plane after a full stack restart.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/43b87133cd3a84ec. Report an issue: GitHub.