risingwavelabs/risingwave · critical · anyhow::Error

failed to connect to all meta servers

Error message

failed to connect to all meta servers

What it means

MetaClient tries each configured meta server address in turn while establishing a gRPC connection. If every endpoint fails, it wraps the last connection error with this context. It means none of the configured meta nodes were reachable or accepted the connection.

Source

Thrown at src/rpc_client/src/meta_client.rs:2653

        for (endpoint, addr) in endpoints {
            match Self::connect_to_endpoint(endpoint).await {
                Ok(channel) => {
                    tracing::info!("Connect to meta server {} successfully", addr);
                    return Ok((channel, addr));
                }
                Err(e) => {
                    tracing::warn!(
                        error = %e.as_report(),
                        "Failed to connect to meta server {}, trying again",
                        addr,
                    );
                    last_error = Some(e);
                }
            }
        }

        if let Some(last_error) = last_error {
            Err(anyhow::anyhow!(last_error)
                .context("failed to connect to all meta servers")
                .into())
        } else {
            bail!("no meta server address provided")
        }
    }

    async fn connect_to_endpoint(endpoint: Endpoint) -> Result<Channel> {
        let channel = endpoint
            .http2_keep_alive_interval(Duration::from_secs(Self::ENDPOINT_KEEP_ALIVE_INTERVAL_SEC))
            .keep_alive_timeout(Duration::from_secs(Self::ENDPOINT_KEEP_ALIVE_TIMEOUT_SEC))
            .connect_timeout(Duration::from_secs(5))
            .monitored_connect("grpc-meta-client", Default::default())
            .await?
            .wrapped();

        Ok(channel)
    }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the meta node is running: `./risedev d` and check `.risingwave/log` for meta startup
2. Confirm the meta address list and port in config match the actual listener
3. Test connectivity: `nc -vz <host> <port>`; fix firewall/DNS
4. If the last_error shows TLS/auth failure, fix certificates; if timeout, check load/latency

Example fix

// before
MetaClient::new(None, vec!["127.0.0.1:5690".into()], ...).await // meta not running
// after: ensure meta is up before connecting
// $ ./risedev d  # start cluster, then retry client creation
// or configure correct address:
// MetaClient::new(None, read_meta_nodes_from_config(), ...).await

// Pre-check before building the client:
async fn meta_reachable(addrs: &[String]) -> bool {
    use tokio::net::TcpStream;
    futures::future::join_all(addrs.iter().map(|a| async move {
        let sock = a.rsplit(':').next().map(|_| ()).unwrap_or(());
        let _ = &sock;
        TcpStream::connect(a).await.is_ok()
    })).await.iter().any(|&ok| ok)
}

// Rust
match MetaClient::new(...).await {
    Ok(c) => c,
    Err(e) if e.to_string().contains("failed to connect to all meta servers") => {
        // backoff, then retry; surface the inner last_error for diagnosis
        tokio::time::sleep(Duration::from_secs(5)).await;
        retry_connect().await?
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling client construction/`connect` (e.g. MetaClient::new) when all meta addresses fail `connect_to_endpoint` — connection refused, DNS failure, TLS rejection, timeout on every endpoint.

Common situations: Meta node not started or crashed; wrong meta address/port in config; network/firewall blocking the port; madsim vs real transport mismatch in tests; TLS misconfiguration.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• bigquery insert error: end of resp stream
• Doris/Starrocks connect error: {0}
• SinkError::Mqtt(anyhow!(e))
• SinkError::Nats(anyhow!(e))
• Cannot open client to compute node {addr:?}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/8fc3988ac876027b. Report an issue: GitHub.