{"record":{"id":"229307dbb9eeac8d","repo":"FuelLabs/fuel-core","slug":"cannot-access-epoch-token-poisoned-lock","errorCode":null,"errorMessage":"cannot access epoch token, poisoned lock","messagePattern":"cannot access epoch token, poisoned lock","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":952,"sourceCode":"            tracing::debug!(\n                quorum = self.quorum,\n                redis_nodes = self.redis_nodes.len(),\n                current_epoch_token = ?self.current_epoch_token_value(),\n                lease_key = %self.lease_key,\n                \"This authority already holds leader lock quorum\"\n            );\n            return Ok(true);\n        }\n        self.acquire_lease_if_free().await\n    }\n\n    async fn release_if_owner(&self) -> anyhow::Result<()> {\n        tracing::debug!(\"Releasing Redis leader lock\");\n        if !self.has_lease_owner_quorum().await? {\n            let mut current_epoch_token = self\n                .current_epoch_token\n                .lock()\n                .map_err(|_| anyhow!(\"cannot access epoch token, poisoned lock\"))?;\n            *current_epoch_token = None;\n            return Ok(());\n        }\n\n        let releases = futures::future::join_all(\n            self.redis_nodes\n                .iter()\n                .map(|redis_node| self.release_lease_on_node(redis_node)),\n        )\n        .await;\n        let released_count = releases.into_iter().filter(|released| *released).count();\n        if self.quorum_reached(released_count) {\n            let mut current_epoch_token = self\n                .current_epoch_token\n                .lock()\n                .map_err(|_| anyhow!(\"cannot access epoch token, poisoned lock\"))?;\n            *current_epoch_token = None;\n            Ok(())","sourceCodeStart":934,"sourceCodeEnd":970,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L934-L970","documentation":"release_if_owner (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:952): when quorum ownership is not held, the adapter clears the shared current_epoch_token (std Mutex<Option<u64>>) so a future leader re-promotes from scratch. lock() returned a PoisonError — a thread panicked earlier while holding this mutex — so the token can be neither read nor written in this process and even the release path fails.","triggerScenarios":"Any earlier panic in code holding current_epoch_token (release, acquire, or epoch adoption); every subsequent lock on it fails, including this one during shutdown or leadership loss. The Redis lease itself still expires via TTL, but in-process epoch state is frozen.","commonSituations":"Seen after an unrelated panic in the PoA service; the node keeps hitting this on every release attempt until restarted; the drop_release_guard may still fire a best-effort release on drop.","solutions":["Restart the fuel-core process — poisoned std Mutexes do not recover in-process","Find and fix the original panic earlier in the logs; this error is only its aftershock","Run under a supervisor with auto-restart; the lease TTL expiry hands leadership to another node safely in the meantime"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_poisoned_lock(e: &anyhow::Error) -> bool {\n    e.to_string().contains(\"poisoned\")\n}","tryCatchPattern":"match leader_lock.release_if_owner().await {\n    Err(e) if is_poisoned_lock(&e) => {\n        // poison is terminal in-process: exit and let the supervisor restart;\n        // lease TTL expiry releases leadership safely even if release failed\n        tracing::error!(\"epoch mutex poisoned on release; exiting for restart\");\n        std::process::exit(1);\n    }\n    other => other,\n}","preventionTips":["Supervise the process with auto-restart; TTL-based leases make restart safe","Treat any panic inside the poa service as page-worthy — poison follows it","Keep the leader-lock TTL short enough that a crashed node loses leadership before the next block is due"],"tags":["fuel-core","poa","mutex","panic","shutdown"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}