What causes "Cannot open client to compute node {addr:?}" in risingwavelabs/risingwave?

Verify the failing compute node is running and healthy (check its logs / process)

Cannot open client to compute node {addr:?}

Error message

Cannot open client to compute node {addr:?}

What it means

The resize_cache ctl command connects to every worker (compute) node to issue a ResizeCache RPC. If establishing the gRPC client to a node fails, it panics with this message instead of continuing, because resizing requires every compute node to be reachable.

Source

Thrown at src/ctl/src/cmd_impl/hummock/resize_cache.rs:53

    meta_cache_capacity: Option<u64>,
    data_cache_capacity: Option<u64>,
    clear_meta_cache: bool,
    clear_data_cache: bool,
) -> anyhow::Result<()> {
    let meta_client = context.meta_client().await?;

    let GetClusterInfoResponse { worker_nodes, .. } = match meta_client.get_cluster_info().await {
        Ok(resp) => resp,
        Err(e) => {
            fail!("Failed to get cluster info: {}", e.as_report());
        }
    };

    let futures = worker_nodes.iter().map(|worker| async {
        let addr = worker.get_host().expect("worker host must be set");
        let client = ComputeClient::new(addr.into(), &RpcClientConfig::default())
            .await
            .unwrap_or_else(|_| panic!("Cannot open client to compute node {addr:?}"));
        client
            .resize_cache(ResizeCacheRequest {
                meta_cache_capacity: meta_cache_capacity.unwrap_or(0),
                data_cache_capacity: data_cache_capacity.unwrap_or(0),
                clear_meta_cache,
                clear_data_cache,
            })
            .await
    });

    if let Err(e) = try_join_all(futures).await {
        fail!("Failed to resize cache: {}", e.as_report())
    }

    Ok(())
}

View on GitHub (pinned to 6469eb736d)

Solutions

Verify the failing compute node is running and healthy (check its logs / process)
Compare the registered host address with the node's actual reachable address; fix worker registration or DNS
Ensure the RPC port is reachable from where you run the ctl command (firewall/security group)
Restart or re-register the stale worker node, then rerun resize_cache

Example fix

null

Defensive patterns

Strategy: retry

Validate before calling

for addr in worker_addrs {
    if tokio::net::TcpStream::connect(addr).await.is_err() {
        eprintln!("compute node {} unreachable; fix before resize", addr);
    }
}

Try / catch

match ComputeClient::new(addr.into(), &RpcClientConfig::default()).await {
    Ok(client) => { /* resize */ }
    Err(e) => eprintln!("skipping node {addr:?}: {e}; ensure node is up and port reachable"),
}

Prevention

Verify all worker nodes are healthy before cluster-wide operations
Cross-check registered host addresses against actual network endpoints
Open firewall/security-group paths for compute RPC ports

When it happens

Trigger: Compute node is down, wrong host/port in the worker's registered host address, network partition, or TLS config mismatch when ComputeClient::new is called during resize_cache.

Common situations: Running resize against a cluster where a compute node crashed or was decommissioned but still registered in meta; firewall blocking the node's RPC port; DNS resolution failure in container environments.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/74f91f7d62b798d6. Report an issue: GitHub.