{"record":{"id":"43b87133cd3a84ec","repo":"neondatabase/neon","slug":"exhausted-all-attempts-to-retrieve-the-config-from","errorCode":null,"errorMessage":"Exhausted all attempts to retrieve the config from the control plane","messagePattern":"Exhausted all attempts to retrieve the config from the control plane","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"compute_tools/src/spec.rs","lineNumber":132,"sourceCode":"                    Err(anyhow!(msg))\n                } else {\n                    bail!(msg);\n                }\n            }\n        };\n\n        if let Err(e) = &result {\n            error!(\"attempt {} to get config failed with: {}\", attempt, e);\n        } else {\n            return result;\n        }\n\n        attempt += 1;\n        std::thread::sleep(std::time::Duration::from_millis(100));\n    }\n\n    // All attempts failed, return error.\n    Err(anyhow::anyhow!(\n        \"Exhausted all attempts to retrieve the config from the control plane\"\n    ))\n}\n\n/// Check `pg_hba.conf` and update if needed to allow external connections.\npub fn update_pg_hba(pgdata_path: &Path, databricks_pg_hba: Option<&String>) -> Result<()> {\n    // XXX: consider making it a part of config.json\n    let pghba_path = pgdata_path.join(\"pg_hba.conf\");\n\n    // Update pg_hba to contains databricks specfic settings before adding neon settings\n    // PG uses the first record that matches to perform authentication, so we need to have\n    // our rules before the default ones from neon.\n    // See https://www.postgresql.org/docs/current/auth-pg-hba-conf.html\n    if let Some(databricks_pg_hba) = databricks_pg_hba {\n        if config::line_in_file(\n            &pghba_path,\n            &format!(\"include_if_exists {}\\n\", *databricks_pg_hba),\n        )? {","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/compute_tools/src/spec.rs#L114-L150","documentation":"get_config_from_control_plane made 3 sequential GET requests (100ms apart) to {base_uri}/compute/api/v2/computes/{compute_id}/spec, authenticated with the NEON_CONTROL_PLANE_TOKEN env var, and every attempt failed with a retryable error: a reqwest send error (DNS failure, connection refused, TLS problem), HTTP 503, or HTTP 502 (a known intermittent gateway issue). Non-retryable statuses such as 404/500 bail out immediately with a different message, so this error specifically indicates connectivity or availability trouble. At initial startup it fails the run; the configurator's periodic loop keeps retrying afterwards.","triggerScenarios":"All three do_control_plane_request attempts return retryable failures: control-plane DNS/service unreachable, connection refused, TLS errors, 502 from the gateway, or 503 during maintenance.","commonSituations":"Dev environments with a wrong --control-plane-uri; Kubernetes NetworkPolicy blocking egress; control-plane redeploy or outage in progress; the compute booting faster than the control plane after a full stack restart.","solutions":["Read the preceding 'attempt N to get config failed with: ...' log lines - they contain the actual reqwest error or HTTP status","Verify reachability: curl -H \"Authorization: Bearer $NEON_CONTROL_PLANE_TOKEN\" $URI/compute/api/v2/computes/$COMPUTE_ID/spec","Fix the control-plane URI, DNS, or network policy so the request can succeed","For transient 502/503 outages no action is needed - the configurator loop re-requests periodically and recovers automatically"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# before booting, prove the spec endpoint answers\ncurl -sS -o /dev/null -w '%{http_code}\\n' \\\n  -H \"Authorization: Bearer $NEON_CONTROL_PLANE_TOKEN\" \\\n  \"$CONTROL_PLANE_URI/compute/api/v2/computes/$COMPUTE_ID/spec\"","typeGuard":null,"tryCatchPattern":"// caller-side retry with backoff on top of the built-in 3 attempts\nlet mut backoff = Duration::from_millis(500);\nloop {\n    match get_config_from_control_plane(uri, &compute_id) {\n        Ok(cfg) => break cfg,\n        Err(e) if retries_left() => {\n            warn!(\"config fetch failed: {e}; retrying in {:?}\", backoff);\n            thread::sleep(backoff);\n            backoff *= 2;\n        }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Validate control-plane URI, DNS and NEON_CONTROL_PLANE_TOKEN as part of environment checks at startup","Retry with backoff at the caller during control-plane deploys - the configurator loop already does this","Watch the 'attempt N to get config failed' lines to distinguish 502/503 (wait it out) from DNS/egress errors (fix config)"],"tags":["control-plane","http","retry","configuration","rust"],"backgroundTag":"control-plane-unreachable","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}