{"record":{"id":"adf88162741d2dc6","repo":"FuelLabs/fuel-core","slug":"failed-to-release-lease-on-quorum","errorCode":null,"errorMessage":"Failed to release lease on quorum","messagePattern":"Failed to release lease on quorum","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":972,"sourceCode":"            return Ok(());\n        }\n\n        let releases = futures::future::join_all(\n            self.redis_nodes\n                .iter()\n                .map(|redis_node| self.release_lease_on_node(redis_node)),\n        )\n        .await;\n        let released_count = releases.into_iter().filter(|released| *released).count();\n        if self.quorum_reached(released_count) {\n            let mut current_epoch_token = self\n                .current_epoch_token\n                .lock()\n                .map_err(|_| anyhow!(\"cannot access epoch token, poisoned lock\"))?;\n            *current_epoch_token = None;\n            Ok(())\n        } else {\n            Err(anyhow!(\"Failed to release lease on quorum\"))\n        }\n    }\n\n    fn current_epoch_token_value(&self) -> Option<u64> {\n        self.current_epoch_token\n            .lock()\n            .ok()\n            .and_then(|epoch| *epoch)\n    }\n\n    fn publish_block_on_all_nodes(\n        &self,\n        epoch: u64,\n        block: &SealedBlock,\n        block_data: &[u8],\n    ) -> Vec<anyhow::Result<WriteBlockResult>> {\n        // Detached `std::thread::spawn` (not scoped) lets us return as soon\n        // as `Written` quorum is reached without waiting for slow nodes.","sourceCodeStart":954,"sourceCodeEnd":990,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L954-L990","documentation":"release_if_owner releases the leader lease on every configured Redis node via release_lease_on_node, counts confirmations, and clears the local epoch token only when released_count >= self.quorum (quorum_reached, poa.rs:369). When fewer than quorum nodes confirm release, it returns this error and deliberately keeps the local token set, so the node still considers itself leader until the lease TTL expires.","triggerScenarios":"More than (redis_nodes.len() - quorum) nodes return false from release_lease_on_node — connection error, Lua release script error (poa.rs:362-365 clears cached connections on Err), or the node no longer holds the key. Example: 5 Redis nodes, quorum 3, 3 nodes unreachable → only 2 releases.","commonSituations":"Network partition between the fuel node and part of the Redis quorum during shutdown or leader step-down; Redis endpoints restarted or overloaded; a quorum configured larger than the number of reachable nodes.","solutions":["Check connectivity and health to every configured Redis endpoint (redis-cli ping) and restore the unreachable ones.","Verify the configured quorum is <= the number of configured Redis nodes and matches the intended failure tolerance.","If endpoints cannot be restored quickly, rely on lease TTL: the lease self-expires and the next leader acquires it — safe by design.","Once connectivity is healthy, retry the step-down/release."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-flight before triggering step-down: confirm a quorum of Redis\n// endpoints answers PING, so release can realistically succeed.\nasync fn quorum_reachable(redis_nodes: &[RedisNode], quorum: usize) -> bool {\n    let oks = futures::future::join_all(\n        redis_nodes.iter().map(|n| async { n.client().ping().await.is_ok() }),\n    ).await;\n    oks.into_iter().filter(|ok| *ok).count() >= quorum\n}","typeGuard":null,"tryCatchPattern":"// On failure: log a warning and let the lease TTL handle safety — do NOT\n// block shutdown on quorum release.\nif let Err(e) = release_result {\n    if e.to_string().contains(\"Failed to release lease on quorum\") {\n        tracing::warn!(\"lease release fell short of quorum; waiting for TTL expiry\");\n    } else {\n        return Err(e);\n    }\n}","preventionTips":["Monitor Redis endpoint health from the fuel node; alert before availability drops below quorum.","Size quorum against expected node failures (quorum <= nodes - tolerated_failures).","Pick lease TTL short enough that a failed release self-heals quickly."],"tags":["redis","quorum","lease","leader-election","network"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}