FuelLabs/fuel-core · error · anyhow::Error

Timed out while connecting to redis leader-lock node

Error message

Timed out while connecting to redis leader-lock node

What it means

multiplexed_connection (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:268) wraps redis get_multiplexed_async_connection in tokio timeout(node_timeout). If the TCP connect/handshake to a Redis node does not complete within node_timeout, the elapsed branch maps to this error; nothing is cached, and callers like check_lease_owner_on_node simply count the node as down for the current round.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:268

        self
    }

    async fn multiplexed_connection(
        &self,
        redis_node: &RedisNode,
    ) -> anyhow::Result<redis::aio::MultiplexedConnection> {
        if let Some(connection) =
            redis_node.cached_connection.lock().await.as_ref().cloned()
        {
            return Ok(connection);
        }

        let new_connection = timeout(
            self.node_timeout,
            redis_node.redis_client.get_multiplexed_async_connection(),
        )
        .await
        .map_err(|_| anyhow!("Timed out while connecting to redis leader-lock node"))??;
        let mut cached_connection = redis_node.cached_connection.lock().await;
        if let Some(connection) = cached_connection.as_ref().cloned() {
            return Ok(connection);
        }
        *cached_connection = Some(new_connection.clone());
        Ok(new_connection)
    }

    async fn clear_cached_connection(&self, redis_node: &RedisNode) {
        let mut cached_connection = redis_node.cached_connection.lock().await;
        *cached_connection = None;
        poa_metrics().connection_reset_total.inc();
    }

    async fn check_lease_owner_on_node(&self, redis_node: &RedisNode) -> bool {
        let mut connection = match self.multiplexed_connection(redis_node).await {
            Ok(connection) => connection,
            Err(_) => return false,

View on GitHub (pinned to b9d4d170da)

Solutions

1. Verify reachability from the fuel-node host: redis-cli -u <url> PING
2. Fix the URL/port or open the network path (security groups, firewall rules)
3. Increase node_timeout so it comfortably exceeds connect RTT plus Redis handshake
4. Rely on quorum meanwhile: keep at least quorum healthy nodes so leadership operations continue while the node recovers

Example fix

// before (consensus/leader-lock config): node_timeout: 200ms
// after:                                 node_timeout: 3s

// pre-flight from the fuel host before starting the node:
// for url in redis_urls: redis-cli -u $url PING must return PONG
let ok = redis::Client::open(url.clone())
    .and_then(|c| c.get_connection())
    .is_ok();

// the adapter itself treats a failed node as non-owner and relies on quorum;
// at the call site, count failures and alert only when they threaten quorum:
if connect_failures >= total_nodes - quorum + 1 {
    alert!("redis leader-lock quorum at risk");
}

When it happens

Trigger: Any leader-lock Redis node unreachable or too slow to complete the connection handshake within node_timeout: connection refused, network partition, firewall silently dropping SYNs, DNS stall, or severe node overload.

Common situations: Redis pod down or mid-restart on Kubernetes; wrong URL/port or blocked security group; ElastiCache failover in progress; node_timeout configured aggressively low relative to network RTT.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• At least one redis url is required for leader lock
• Timed out reading latest stream entry from Redis node
• Timed out reading stream entries from Redis node
• Backlog unresolved at height {current_height}: repair error:
• Failed to release lease on quorum

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/b0a1425cdafcd71a. Report an issue: GitHub.