{"record":{"id":"8fc3988ac876027b","repo":"risingwavelabs/risingwave","slug":"failed-to-connect-to-all-meta-servers","errorCode":null,"errorMessage":"failed to connect to all meta servers","messagePattern":"failed to connect to all meta servers","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"src/rpc_client/src/meta_client.rs","lineNumber":2653,"sourceCode":"        for (endpoint, addr) in endpoints {\n            match Self::connect_to_endpoint(endpoint).await {\n                Ok(channel) => {\n                    tracing::info!(\"Connect to meta server {} successfully\", addr);\n                    return Ok((channel, addr));\n                }\n                Err(e) => {\n                    tracing::warn!(\n                        error = %e.as_report(),\n                        \"Failed to connect to meta server {}, trying again\",\n                        addr,\n                    );\n                    last_error = Some(e);\n                }\n            }\n        }\n\n        if let Some(last_error) = last_error {\n            Err(anyhow::anyhow!(last_error)\n                .context(\"failed to connect to all meta servers\")\n                .into())\n        } else {\n            bail!(\"no meta server address provided\")\n        }\n    }\n\n    async fn connect_to_endpoint(endpoint: Endpoint) -> Result<Channel> {\n        let channel = endpoint\n            .http2_keep_alive_interval(Duration::from_secs(Self::ENDPOINT_KEEP_ALIVE_INTERVAL_SEC))\n            .keep_alive_timeout(Duration::from_secs(Self::ENDPOINT_KEEP_ALIVE_TIMEOUT_SEC))\n            .connect_timeout(Duration::from_secs(5))\n            .monitored_connect(\"grpc-meta-client\", Default::default())\n            .await?\n            .wrapped();\n\n        Ok(channel)\n    }","sourceCodeStart":2635,"sourceCodeEnd":2671,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/rpc_client/src/meta_client.rs#L2635-L2671","documentation":"MetaClient tries each configured meta server address in turn while establishing a gRPC connection. If every endpoint fails, it wraps the last connection error with this context. It means none of the configured meta nodes were reachable or accepted the connection.","triggerScenarios":"Calling client construction/`connect` (e.g. MetaClient::new) when all meta addresses fail `connect_to_endpoint` — connection refused, DNS failure, TLS rejection, timeout on every endpoint.","commonSituations":"Meta node not started or crashed; wrong meta address/port in config; network/firewall blocking the port; madsim vs real transport mismatch in tests; TLS misconfiguration.","solutions":["Verify the meta node is running: `./risedev d` and check `.risingwave/log` for meta startup","Confirm the meta address list and port in config match the actual listener","Test connectivity: `nc -vz <host> <port>`; fix firewall/DNS","If the last_error shows TLS/auth failure, fix certificates; if timeout, check load/latency"],"exampleFix":"// before\nMetaClient::new(None, vec![\"127.0.0.1:5690\".into()], ...).await // meta not running\n// after: ensure meta is up before connecting\n// $ ./risedev d  # start cluster, then retry client creation\n// or configure correct address:\n// MetaClient::new(None, read_meta_nodes_from_config(), ...).await","handlingStrategy":"retry","validationCode":"// Pre-check before building the client:\nasync fn meta_reachable(addrs: &[String]) -> bool {\n    use tokio::net::TcpStream;\n    futures::future::join_all(addrs.iter().map(|a| async move {\n        let sock = a.rsplit(':').next().map(|_| ()).unwrap_or(());\n        let _ = &sock;\n        TcpStream::connect(a).await.is_ok()\n    })).await.iter().any(|&ok| ok)\n}","typeGuard":null,"tryCatchPattern":"// Rust\nmatch MetaClient::new(...).await {\n    Ok(c) => c,\n    Err(e) if e.to_string().contains(\"failed to connect to all meta servers\") => {\n        // backoff, then retry; surface the inner last_error for diagnosis\n        tokio::time::sleep(Duration::from_secs(5)).await;\n        retry_connect().await?\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Always start the meta service before clients","Validate meta endpoints in config with a TCP probe at boot","Provide multiple meta addresses for redundancy","Monitor meta node liveness in production"],"tags":["network","grpc","connection","meta-service"],"backgroundTag":"connection-refused","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}