What causes "Cannot open client to compute node {addr:?}" in risingwavelabs/risingwave?

Verify the failing compute node is running and healthy (check its logs / process)

Cannot open client to compute node

Error message

Cannot open client to compute node {addr:?}

What it means

The resize_cache ctl command connects to every worker (compute) node to issue a ResizeCache RPC. If establishing the gRPC client to a node fails, it panics with this message instead of continuing, because resizing requires every compute node to be reachable.

Solutions

Verify the failing compute node is running and healthy (check its logs / process)
Compare the registered host address with the node's actual reachable address; fix worker registration or DNS
Ensure the RPC port is reachable from where you run the ctl command (firewall/security group)
Restart or re-register the stale worker node, then rerun resize_cache

Example fix

null

Defensive patterns

Strategy: retry

Validate before calling

for addr in worker_addrs {
    if tokio::net::TcpStream::connect(addr).await.is_err() {
        eprintln!("compute node {} unreachable; fix before resize", addr);
    }
}

Try / catch

match ComputeClient::new(addr.into(), &RpcClientConfig::default()).await {
    Ok(client) => { /* resize */ }
    Err(e) => eprintln!("skipping node {addr:?}: {e}; ensure node is up and port reachable"),
}

Prevention

Verify all worker nodes are healthy before cluster-wide operations
Cross-check registered host addresses against actual network endpoints
Open firewall/security-group paths for compute RPC ports

When it happens

Trigger: Compute node is down, wrong host/port in the worker's registered host address, network partition, or TLS config mismatch when ComputeClient::new is called during resize_cache.

Common situations: Running resize against a cluster where a compute node crashed or was decommissioned but still registered in meta; firewall blocking the node's RPC port; DNS resolution failure in container environments.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/74f91f7d62b798d6. Report an issue: GitHub.

Appendix: source

Thrown at src/ctl/src/cmd_impl/hummock/resize_cache.rs:53

    meta_cache_capacity: Option<u64>,
    data_cache_capacity: Option<u64>,
    clear_meta_cache: bool,
    clear_data_cache: bool,
) -> anyhow::Result<()> {
    let meta_client = context.meta_client().await?;

    let GetClusterInfoResponse { worker_nodes, .. } = match meta_client.get_cluster_info().await {
        Ok(resp) => resp,
        Err(e) => {
            fail!("Failed to get cluster info: {}", e.as_report());
        }
    };

    let futures = worker_nodes.iter().map(|worker| async {
        let addr = worker.get_host().expect("worker host must be set");
        let client = ComputeClient::new(addr.into(), &RpcClientConfig::default())
            .await
            .unwrap_or_else(|_| panic!("Cannot open client to compute node {addr:?}"));
        client
            .resize_cache(ResizeCacheRequest {
                meta_cache_capacity: meta_cache_capacity.unwrap_or(0),
                data_cache_capacity: data_cache_capacity.unwrap_or(0),
                clear_meta_cache,
                clear_data_cache,
            })
            .await
    });

    if let Err(e) = try_join_all(futures).await {
        fail!("Failed to resize cache: {}", e.as_report())
    }

    Ok(())
}

View on GitHub (pinned to 6469eb736d)