{"record":{"id":"b0a1425cdafcd71a","repo":"FuelLabs/fuel-core","slug":"timed-out-while-connecting-to-redis-leader-lock-no","errorCode":null,"errorMessage":"Timed out while connecting to redis leader-lock node","messagePattern":"Timed out while connecting to redis leader-lock node","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":268,"sourceCode":"        self\n    }\n\n    async fn multiplexed_connection(\n        &self,\n        redis_node: &RedisNode,\n    ) -> anyhow::Result<redis::aio::MultiplexedConnection> {\n        if let Some(connection) =\n            redis_node.cached_connection.lock().await.as_ref().cloned()\n        {\n            return Ok(connection);\n        }\n\n        let new_connection = timeout(\n            self.node_timeout,\n            redis_node.redis_client.get_multiplexed_async_connection(),\n        )\n        .await\n        .map_err(|_| anyhow!(\"Timed out while connecting to redis leader-lock node\"))??;\n        let mut cached_connection = redis_node.cached_connection.lock().await;\n        if let Some(connection) = cached_connection.as_ref().cloned() {\n            return Ok(connection);\n        }\n        *cached_connection = Some(new_connection.clone());\n        Ok(new_connection)\n    }\n\n    async fn clear_cached_connection(&self, redis_node: &RedisNode) {\n        let mut cached_connection = redis_node.cached_connection.lock().await;\n        *cached_connection = None;\n        poa_metrics().connection_reset_total.inc();\n    }\n\n    async fn check_lease_owner_on_node(&self, redis_node: &RedisNode) -> bool {\n        let mut connection = match self.multiplexed_connection(redis_node).await {\n            Ok(connection) => connection,\n            Err(_) => return false,","sourceCodeStart":250,"sourceCodeEnd":286,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L250-L286","documentation":"multiplexed_connection (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:268) wraps redis get_multiplexed_async_connection in tokio timeout(node_timeout). If the TCP connect/handshake to a Redis node does not complete within node_timeout, the elapsed branch maps to this error; nothing is cached, and callers like check_lease_owner_on_node simply count the node as down for the current round.","triggerScenarios":"Any leader-lock Redis node unreachable or too slow to complete the connection handshake within node_timeout: connection refused, network partition, firewall silently dropping SYNs, DNS stall, or severe node overload.","commonSituations":"Redis pod down or mid-restart on Kubernetes; wrong URL/port or blocked security group; ElastiCache failover in progress; node_timeout configured aggressively low relative to network RTT.","solutions":["Verify reachability from the fuel-node host: redis-cli -u <url> PING","Fix the URL/port or open the network path (security groups, firewall rules)","Increase node_timeout so it comfortably exceeds connect RTT plus Redis handshake","Rely on quorum meanwhile: keep at least quorum healthy nodes so leadership operations continue while the node recovers"],"exampleFix":"// before (consensus/leader-lock config): node_timeout: 200ms\n// after:                                 node_timeout: 3s","handlingStrategy":"retry","validationCode":"// pre-flight from the fuel host before starting the node:\n// for url in redis_urls: redis-cli -u $url PING must return PONG\nlet ok = redis::Client::open(url.clone())\n    .and_then(|c| c.get_connection())\n    .is_ok();","typeGuard":null,"tryCatchPattern":"// the adapter itself treats a failed node as non-owner and relies on quorum;\n// at the call site, count failures and alert only when they threaten quorum:\nif connect_failures >= total_nodes - quorum + 1 {\n    alert!(\"redis leader-lock quorum at risk\");\n}","preventionTips":["Health-check every redis url (PING) before node start and periodically","Size node_timeout above worst-case connect RTT (include TLS and DNS time)","Keep >= quorum nodes healthy during maintenance windows; never restart all at once"],"tags":["fuel-core","redis","network","timeout","leader-election"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}