{"record":{"id":"9f9caa49e1da957a","repo":"FuelLabs/fuel-core","slug":"cannot-reconcile-only-redis-nodes-responded","errorCode":null,"errorMessage":"Cannot reconcile: only {}/{} Redis nodes responded ({} failed)","messagePattern":"Cannot reconcile: only (.+?)/(.+?) Redis nodes responded \\((.+?) failed\\)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":676,"sourceCode":"            match result {\n                Ok(Some((latest_height, _latest_stream_id))) => {\n                    successful_reads = successful_reads.saturating_add(1);\n                    if latest_height >= next_height {\n                        nodes_indicating_backlog =\n                            nodes_indicating_backlog.saturating_add(1);\n                    }\n                }\n                Ok(None) => {\n                    successful_reads = successful_reads.saturating_add(1);\n                }\n                Err(e) => {\n                    tracing::warn!(\"Redis latest stream read failed: {e}\");\n                    failed_count = failed_count.saturating_add(1);\n                }\n            }\n        }\n        if !self.quorum_reached(successful_reads) {\n            return Err(anyhow!(\n                \"Cannot reconcile: only {}/{} Redis nodes responded ({} failed)\",\n                successful_reads,\n                self.redis_nodes.len(),\n                failed_count\n            ));\n        }\n        Ok(nodes_indicating_backlog > 0)\n    }\n\n    async fn read_stream_entries_on_node(\n        &self,\n        redis_node: &RedisNode,\n        next_height: u32,\n        max_entries: usize,\n    ) -> anyhow::Result<Vec<(u32, u64, SealedBlock)>> {\n        if max_entries == 0 {\n            return Ok(Vec::new());\n        }","sourceCodeStart":658,"sourceCodeEnd":694,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L658-L694","documentation":"should_reconcile_from_stream (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:676) asks every Redis node for its latest stream entry and counts successful reads (Ok(Some) and Ok(None) both count). If fewer than quorum (majority + disruption budget) responded, the node cannot safely decide whether a backlog exists and refuses to reconcile, failing the block-production path that triggered the check.","triggerScenarios":"More than nodes - quorum Redis nodes failing latest-entry reads — timeouts, protocol errors, or connection failures (each logged as 'Redis latest stream read failed' just before) — while the node is preparing to produce at next_height.","commonSituations":"Majority Redis outage or rolling restart; network partition between fuel node and Redis cluster; systemic timeouts under load; redis_urls listing dead nodes that were never pruned from config.","solutions":["Restore Redis availability until at least quorum nodes answer; the per-node warnings name the failing cause","Verify network path and URLs from the fuel host (PING each URL)","Raise node_timeout or reduce Redis load if nodes are slow rather than down","If nodes were permanently removed, update redis_urls so quorum arithmetic matches the real cluster"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before relying on reconciliation, verify quorum reachability:\nlet healthy = ping_all(redis_urls, node_timeout).await; // count PONGs\nanyhow::ensure!(healthy >= quorum,\n    \"only {healthy} redis nodes reachable; quorum is {quorum}\");","typeGuard":null,"tryCatchPattern":"match should_reconcile(next_height).await {\n    Err(e) if e.to_string().starts_with(\"Cannot reconcile: only\") => {\n        // availability failure: backoff, restore nodes, retry next production round\n        tokio::time::sleep(backoff).await;\n    }\n    other => other,\n}","preventionTips":["Run an odd number >= 3 of Redis nodes across failure domains","Continuous external health checks on all urls; page when reachable count approaches quorum","Keep redis_urls in config in sync with the actual cluster membership"],"tags":["fuel-core","redis","quorum","availability","reconciliation"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}