FuelLabs/fuel-core · error · anyhow::Error

Cannot reconcile: only

Error message

Cannot reconcile: only {}/{} Redis nodes responded ({} failed)

What it means

should_reconcile_from_stream (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:676) asks every Redis node for its latest stream entry and counts successful reads (Ok(Some) and Ok(None) both count). If fewer than quorum (majority + disruption budget) responded, the node cannot safely decide whether a backlog exists and refuses to reconcile, failing the block-production path that triggered the check.

Solutions

1. Restore Redis availability until at least quorum nodes answer; the per-node warnings name the failing cause
2. Verify network path and URLs from the fuel host (PING each URL)
3. Raise node_timeout or reduce Redis load if nodes are slow rather than down
4. If nodes were permanently removed, update redis_urls so quorum arithmetic matches the real cluster

// before relying on reconciliation, verify quorum reachability:
let healthy = ping_all(redis_urls, node_timeout).await; // count PONGs
anyhow::ensure!(healthy >= quorum,
    "only {healthy} redis nodes reachable; quorum is {quorum}");

match should_reconcile(next_height).await {
    Err(e) if e.to_string().starts_with("Cannot reconcile: only") => {
        // availability failure: backoff, restore nodes, retry next production round
        tokio::time::sleep(backoff).await;
    }
    other => other,
}

When it happens

Trigger: More than nodes - quorum Redis nodes failing latest-entry reads — timeouts, protocol errors, or connection failures (each logged as 'Redis latest stream read failed' just before) — while the node is preparing to produce at next_height.

Common situations: Majority Redis outage or rolling restart; network partition between fuel node and Redis cluster; systemic timeouts under load; redis_urls listing dead nodes that were never pruned from config.

Related errors

• Backlog unresolved at height
• Backlog unresolved at height
• Backlog unresolved at height
• Failed to publish block to redis quorum with fencing checks
• Timed out reading stream entries from Redis node

AI-assisted analysis of FuelLabs/fuel-core@add100d30d (2026-09-05). Data as JSON: /api/errors/9f9caa49e1da957a. Report an issue: GitHub.

Appendix: source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:816

                return Ok(true);
            }
            if quorum_shortcircuit && self.quorum_reached(no_backlog_reads) {
                poa_metrics().observe_quorum_latency(
                    QuorumOp::ReconcileRead,
                    quorum_start.elapsed().as_secs_f64(),
                );
                drop(reads);
                return Ok(false);
            }
        }
        if self.quorum_reached(successful_reads) {
            poa_metrics().observe_quorum_latency(
                QuorumOp::ReconcileRead,
                quorum_start.elapsed().as_secs_f64(),
            );
            Ok(false)
        } else {
            Err(anyhow!(
                "Cannot reconcile: only {}/{} Redis nodes responded ({} failed)",
                successful_reads,
                n,
                failed_count
            ))
        }
    }

    async fn read_stream_entries_on_node(
        &self,
        redis_node: &RedisNode,
        next_height: u32,
        max_entries: usize,
    ) -> anyhow::Result<Vec<(u32, u64, SealedBlock)>> {
        if max_entries == 0 {
            return Ok(Vec::new());
        }

View on GitHub (pinned to add100d30d)