{"record":{"id":"7c9286fcc3121f77","repo":"FuelLabs/fuel-core","slug":"failed-to-publish-block-to-redis-quorum-with-fenci","errorCode":null,"errorMessage":"Failed to publish block to redis quorum with fencing checks","messagePattern":"Failed to publish block to redis quorum with fencing checks","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":1394,"sourceCode":"        };\n        let block_data = postcard::to_allocvec(block)?;\n        let successes = self\n            .publish_block_on_all_nodes(epoch, block, &block_data)\n            .into_iter()\n            .map(|result| match result {\n                Ok(WriteBlockResult::Written) => true,\n                Ok(_) => false,\n                Err(err) => {\n                    tracing::debug!(\"Redis publish on node failed: {err}\");\n                    false\n                }\n            })\n            .filter(|success| *success)\n            .count();\n        if self.quorum_reached(successes) {\n            Ok(())\n        } else {\n            Err(anyhow!(\n                \"Failed to publish block to redis quorum with fencing checks\"\n            ))\n        }\n    }\n}\n\n#[async_trait::async_trait]\nimpl ConsensusModulePort for PoAAdapter {\n    async fn manually_produce_blocks(\n        &self,\n        start_time: Option<Tai64>,\n        number_of_blocks: u32,\n    ) -> anyhow::Result<()> {\n        self.manually_produce_blocks(start_time, Mode::Blocks { number_of_blocks })\n            .await\n    }\n}\n","sourceCodeStart":1376,"sourceCodeEnd":1412,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L1376-L1412","documentation":"After fanning write_block.lua out to all Redis nodes, publish_produced_block counts only Ok(Written) results and requires quorum_reached(successes). Fewer than quorum Written means the block is NOT durably published: transport errors (logged at debug as 'Redis publish on node failed'), HeightExists (some other block already at that height), and FencingRejected (lease lost) all count as non-success.","triggerScenarios":"Enough Redis nodes down/unreachable to break quorum; lease lost mid-publish (fencing rejections); or a competing leader wrote different blocks at the same height on enough nodes (HeightExists).","commonSituations":"Redis outage or network partition during block production; lease TTL expiring mid-publish; quorum misconfigured above the reachable node count; two leaders racing at the same height.","solutions":["Check Redis cluster health and connectivity from the node; restore unreachable endpoints.","Enable debug logging and distinguish 'Redis publish on node failed' (transport) from fencing rejections (lease lost) in the logs.","Verify quorum setting vs number of configured Redis nodes.","If fencing rejections appear: stop producing, let the current lease holder reconcile, re-acquire the lease before retrying.","Once quorum is healthy, retry producing the same height."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-flight: confirm quorum of Redis endpoints reachable before a\n// production round that must publish.\nasync fn can_publish_to_quorum(nodes: &[RedisNode], quorum: usize) -> bool {\n    let oks = futures::future::join_all(\n        nodes.iter().map(|n| async { n.ping().await.is_ok() }),\n    ).await;\n    oks.into_iter().filter(|ok| *ok).count() >= quorum\n}","typeGuard":null,"tryCatchPattern":"// Publish failure is retryable ONLY after health checks pass and the lease\n// is still held; fencing-related failures must NOT be blindly retried.\nif let Err(e) = publish_result {\n    if e.to_string().contains(\"Failed to publish block to redis quorum\") {\n        // re-verify lease + quorum health, then re-produce this height\n    }\n    return Err(e);\n}","preventionTips":["Monitor Redis quorum availability as a production-blocking health check.","Verify quorum <= number of configured Redis nodes at config load time.","Correlate debug logs ('Redis publish on node failed') with endpoint health before retrying.","Never treat an unpublished block as committed — halt production past that height until resolved."],"tags":["redis","quorum","block-production","fencing","availability"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}