{"record":{"id":"9f9caa49e1da957a","repo":"FuelLabs/fuel-core","slug":"cannot-reconcile-only-redis-nodes-responded","errorCode":null,"errorMessage":"Cannot reconcile: only {}/{} Redis nodes responded ({} failed)","messagePattern":"Cannot reconcile: only (.+?)/(.+?) Redis nodes responded \\((.+?) failed\\)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":816,"sourceCode":"                return Ok(true);\n            }\n            if quorum_shortcircuit && self.quorum_reached(no_backlog_reads) {\n                poa_metrics().observe_quorum_latency(\n                    QuorumOp::ReconcileRead,\n                    quorum_start.elapsed().as_secs_f64(),\n                );\n                drop(reads);\n                return Ok(false);\n            }\n        }\n        if self.quorum_reached(successful_reads) {\n            poa_metrics().observe_quorum_latency(\n                QuorumOp::ReconcileRead,\n                quorum_start.elapsed().as_secs_f64(),\n            );\n            Ok(false)\n        } else {\n            Err(anyhow!(\n                \"Cannot reconcile: only {}/{} Redis nodes responded ({} failed)\",\n                successful_reads,\n                n,\n                failed_count\n            ))\n        }\n    }\n\n    async fn read_stream_entries_on_node(\n        &self,\n        redis_node: &RedisNode,\n        next_height: u32,\n        max_entries: usize,\n    ) -> anyhow::Result<Vec<(u32, u64, SealedBlock)>> {\n        if max_entries == 0 {\n            return Ok(Vec::new());\n        }\n","sourceCodeStart":798,"sourceCodeEnd":834,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/add100d30d21498e8528c46be8567fbd2ea019af/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L798-L834","documentation":"should_reconcile_from_stream (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:676) asks every Redis node for its latest stream entry and counts successful reads (Ok(Some) and Ok(None) both count). If fewer than quorum (majority + disruption budget) responded, the node cannot safely decide whether a backlog exists and refuses to reconcile, failing the block-production path that triggered the check.","triggerScenarios":"More than nodes - quorum Redis nodes failing latest-entry reads — timeouts, protocol errors, or connection failures (each logged as 'Redis latest stream read failed' just before) — while the node is preparing to produce at next_height.","commonSituations":"Majority Redis outage or rolling restart; network partition between fuel node and Redis cluster; systemic timeouts under load; redis_urls listing dead nodes that were never pruned from config.","solutions":["Restore Redis availability until at least quorum nodes answer; the per-node warnings name the failing cause","Verify network path and URLs from the fuel host (PING each URL)","Raise node_timeout or reduce Redis load if nodes are slow rather than down","If nodes were permanently removed, update redis_urls so quorum arithmetic matches the real cluster"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before relying on reconciliation, verify quorum reachability:\nlet healthy = ping_all(redis_urls, node_timeout).await; // count PONGs\nanyhow::ensure!(healthy >= quorum,\n    \"only {healthy} redis nodes reachable; quorum is {quorum}\");","typeGuard":null,"tryCatchPattern":"match should_reconcile(next_height).await {\n    Err(e) if e.to_string().starts_with(\"Cannot reconcile: only\") => {\n        // availability failure: backoff, restore nodes, retry next production round\n        tokio::time::sleep(backoff).await;\n    }\n    other => other,\n}","preventionTips":["Run an odd number >= 3 of Redis nodes across failure domains","Continuous external health checks on all urls; page when reachable count approaches quorum","Keep redis_urls in config in sync with the actual cluster membership"],"tags":["fuel-core","redis","quorum","availability","reconciliation"],"backgroundTag":null,"analyzedSha":"add100d30d21498e8528c46be8567fbd2ea019af","analyzedAt":"2026-09-05T18:46:12.018Z","contentChangedAt":"2026-09-05T18:46:12.018Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}