FuelLabs/fuel-core · critical · anyhow::Error

Failed to publish block to redis quorum with fencing checks

Error message

Failed to publish block to redis quorum with fencing checks

What it means

After fanning write_block.lua out to all Redis nodes, publish_produced_block counts only Ok(Written) results and requires quorum_reached(successes). Fewer than quorum Written means the block is NOT durably published: transport errors (logged at debug as 'Redis publish on node failed'), HeightExists (some other block already at that height), and FencingRejected (lease lost) all count as non-success.

Solutions

Check Redis cluster health and connectivity from the node; restore unreachable endpoints.
Enable debug logging and distinguish 'Redis publish on node failed' (transport) from fencing rejections (lease lost) in the logs.
Verify quorum setting vs number of configured Redis nodes.
If fencing rejections appear: stop producing, let the current lease holder reconcile, re-acquire the lease before retrying.
Once quorum is healthy, retry producing the same height.

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: confirm quorum of Redis endpoints reachable before a
// production round that must publish.
async fn can_publish_to_quorum(nodes: &[RedisNode], quorum: usize) -> bool {
    let oks = futures::future::join_all(
        nodes.iter().map(|n| async { n.ping().await.is_ok() }),
    ).await;
    oks.into_iter().filter(|ok| *ok).count() >= quorum
}

Try / catch

// Publish failure is retryable ONLY after health checks pass and the lease
// is still held; fencing-related failures must NOT be blindly retried.
if let Err(e) = publish_result {
    if e.to_string().contains("Failed to publish block to redis quorum") {
        // re-verify lease + quorum health, then re-produce this height
    }
    return Err(e);
}

Prevention

Monitor Redis quorum availability as a production-blocking health check.
Verify quorum <= number of configured Redis nodes at config load time.
Correlate debug logs ('Redis publish on node failed') with endpoint health before retrying.
Never treat an unpublished block as committed — halt production past that height until resolved.

When it happens

Trigger: Enough Redis nodes down/unreachable to break quorum; lease lost mid-publish (fencing rejections); or a competing leader wrote different blocks at the same height on enough nodes (HeightExists).

Common situations: Redis outage or network partition during block production; lease TTL expiring mid-publish; quorum misconfigured above the reachable node count; two leaders racing at the same height.

Related errors

AI-assisted analysis of FuelLabs/fuel-core@add100d30d (2026-09-05). Data as JSON: /api/errors/7c9286fcc3121f77. Report an issue: GitHub.

Appendix: source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:1642

        let block_data = postcard::to_allocvec(block)?;
        let successes = self
            .publish_block_on_all_nodes(epoch, block, &block_data)
            .await
            .into_iter()
            .map(|result| match result {
                Ok(WriteBlockResult::Written) => true,
                Ok(_) => false,
                Err(err) => {
                    tracing::debug!("Redis publish on node failed: {err}");
                    false
                }
            })
            .filter(|success| *success)
            .count();
        if self.quorum_reached(successes) {
            Ok(())
        } else {
            Err(anyhow!(
                "Failed to publish block to redis quorum with fencing checks"
            ))
        }
    }
}

#[async_trait::async_trait]
impl BlockReconciliationPort for ReconciliationAdapter {
    async fn leader_state(
        &self,
        next_height: BlockHeight,
    ) -> anyhow::Result<LeaderState> {
        match self {
            Self::Redis(adapter) => adapter.leader_state(next_height).await,
            Self::Noop(adapter) => adapter.leader_state(next_height).await,
        }
    }

View on GitHub (pinned to add100d30d)