{"record":{"id":"229307dbb9eeac8d","repo":"FuelLabs/fuel-core","slug":"cannot-access-epoch-token-poisoned-lock","errorCode":null,"errorMessage":"cannot access epoch token, poisoned lock","messagePattern":"cannot access epoch token, poisoned lock","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":1109,"sourceCode":"        // the lease key when the stored owner matches our token, so\n        // calling it on every node is safe regardless of whether we\n        // currently hold quorum. We deliberately do NOT call\n        // `has_lease_owner_quorum` here: that would spawn the\n        // `spawn_acquire_lease_on_unowned_nodes` extension as a\n        // detached task, which can race our own DELs and resurrect\n        // the lease on a node we are simultaneously releasing\n        // (observed as \"Lost lock during repair\" / FENCING_ERROR\n        // when the next leader's writes hit the resurrected entry).\n        let _ = futures::future::join_all(\n            self.redis_nodes\n                .iter()\n                .map(|redis_node| self.release_lease_on_node(redis_node)),\n        )\n        .await;\n        let mut current_epoch_token = self\n            .current_epoch_token\n            .lock()\n            .map_err(|_| anyhow!(\"cannot access epoch token, poisoned lock\"))?;\n        *current_epoch_token = None;\n        Ok(())\n    }\n\n    fn current_epoch_token_value(&self) -> Option<u64> {\n        self.current_epoch_token\n            .lock()\n            .ok()\n            .and_then(|epoch| *epoch)\n    }\n\n    /// Publish `block` to every Redis node in parallel and return as soon\n    /// as `Written` quorum is reached. Each per-node publish runs in its\n    /// own `tokio::spawn`ed task so it completes in the background even\n    /// after we short-circuit — we trade a bit of background work per\n    /// publish for maximum durability: every reachable node gets a real\n    /// write attempt, bounded by `node_timeout`, instead of the call\n    /// being cancelled mid-RESP when the parent drops a `FuturesUnordered`.","sourceCodeStart":1091,"sourceCodeEnd":1127,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/add100d30d21498e8528c46be8567fbd2ea019af/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L1091-L1127","documentation":"release_if_owner (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:952): when quorum ownership is not held, the adapter clears the shared current_epoch_token (std Mutex<Option<u64>>) so a future leader re-promotes from scratch. lock() returned a PoisonError — a thread panicked earlier while holding this mutex — so the token can be neither read nor written in this process and even the release path fails.","triggerScenarios":"Any earlier panic in code holding current_epoch_token (release, acquire, or epoch adoption); every subsequent lock on it fails, including this one during shutdown or leadership loss. The Redis lease itself still expires via TTL, but in-process epoch state is frozen.","commonSituations":"Seen after an unrelated panic in the PoA service; the node keeps hitting this on every release attempt until restarted; the drop_release_guard may still fire a best-effort release on drop.","solutions":["Restart the fuel-core process — poisoned std Mutexes do not recover in-process","Find and fix the original panic earlier in the logs; this error is only its aftershock","Run under a supervisor with auto-restart; the lease TTL expiry hands leadership to another node safely in the meantime"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_poisoned_lock(e: &anyhow::Error) -> bool {\n    e.to_string().contains(\"poisoned\")\n}","tryCatchPattern":"match leader_lock.release_if_owner().await {\n    Err(e) if is_poisoned_lock(&e) => {\n        // poison is terminal in-process: exit and let the supervisor restart;\n        // lease TTL expiry releases leadership safely even if release failed\n        tracing::error!(\"epoch mutex poisoned on release; exiting for restart\");\n        std::process::exit(1);\n    }\n    other => other,\n}","preventionTips":["Supervise the process with auto-restart; TTL-based leases make restart safe","Treat any panic inside the poa service as page-worthy — poison follows it","Keep the leader-lock TTL short enough that a crashed node loses leadership before the next block is due"],"tags":["fuel-core","poa","mutex","panic","shutdown"],"backgroundTag":null,"analyzedSha":"add100d30d21498e8528c46be8567fbd2ea019af","analyzedAt":"2026-09-05T18:46:12.018Z","contentChangedAt":"2026-09-05T18:46:12.018Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}