FuelLabs/fuel-core · error · anyhow::Error

Cannot reconcile: only {}/{} Redis nodes responded ({} faile

Error message

Cannot reconcile: only {}/{} Redis nodes responded ({} failed)

What it means

should_reconcile_from_stream (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:676) asks every Redis node for its latest stream entry and counts successful reads (Ok(Some) and Ok(None) both count). If fewer than quorum (majority + disruption budget) responded, the node cannot safely decide whether a backlog exists and refuses to reconcile, failing the block-production path that triggered the check.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:676

            match result {
                Ok(Some((latest_height, _latest_stream_id))) => {
                    successful_reads = successful_reads.saturating_add(1);
                    if latest_height >= next_height {
                        nodes_indicating_backlog =
                            nodes_indicating_backlog.saturating_add(1);
                    }
                }
                Ok(None) => {
                    successful_reads = successful_reads.saturating_add(1);
                }
                Err(e) => {
                    tracing::warn!("Redis latest stream read failed: {e}");
                    failed_count = failed_count.saturating_add(1);
                }
            }
        }
        if !self.quorum_reached(successful_reads) {
            return Err(anyhow!(
                "Cannot reconcile: only {}/{} Redis nodes responded ({} failed)",
                successful_reads,
                self.redis_nodes.len(),
                failed_count
            ));
        }
        Ok(nodes_indicating_backlog > 0)
    }

    async fn read_stream_entries_on_node(
        &self,
        redis_node: &RedisNode,
        next_height: u32,
        max_entries: usize,
    ) -> anyhow::Result<Vec<(u32, u64, SealedBlock)>> {
        if max_entries == 0 {
            return Ok(Vec::new());
        }

View on GitHub (pinned to b9d4d170da)

Solutions

1. Restore Redis availability until at least quorum nodes answer; the per-node warnings name the failing cause
2. Verify network path and URLs from the fuel host (PING each URL)
3. Raise node_timeout or reduce Redis load if nodes are slow rather than down
4. If nodes were permanently removed, update redis_urls so quorum arithmetic matches the real cluster

// before relying on reconciliation, verify quorum reachability:
let healthy = ping_all(redis_urls, node_timeout).await; // count PONGs
anyhow::ensure!(healthy >= quorum,
    "only {healthy} redis nodes reachable; quorum is {quorum}");

match should_reconcile(next_height).await {
    Err(e) if e.to_string().starts_with("Cannot reconcile: only") => {
        // availability failure: backoff, restore nodes, retry next production round
        tokio::time::sleep(backoff).await;
    }
    other => other,
}

When it happens

Trigger: More than nodes - quorum Redis nodes failing latest-entry reads — timeouts, protocol errors, or connection failures (each logged as 'Redis latest stream read failed' just before) — while the node is preparing to produce at next_height.

Common situations: Majority Redis outage or rolling restart; network partition between fuel node and Redis cluster; systemic timeouts under load; redis_urls listing dead nodes that were never pruned from config.

Related errors

• Backlog unresolved at height {current_height}: repair failed
• Timed out reading stream entries from Redis node
• Backlog unresolved at height {current_height}: stream indica
• Backlog unresolved at height {current_height}: repair error:
• Failed to publish block to redis quorum with fencing checks

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/9f9caa49e1da957a. Report an issue: GitHub.