{"record":{"id":"0058a654a4c19487","repo":"FuelLabs/fuel-core","slug":"epoch-token-lock-poisoned","errorCode":null,"errorMessage":"epoch token lock poisoned: {}","messagePattern":"epoch token lock poisoned: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/fuel-core/src/service/adapters/consensus_module/poa.rs","lineNumber":513,"sourceCode":"            );\n            if self.quorum_reached(acquired_count) && validity_millis > 0 {\n                // Record epoch drift across quorum nodes\n                if promoted_tokens.len() > 1\n                    && let (Some(min_tok), Some(max_tok)) = (\n                        promoted_tokens.iter().copied().min(),\n                        promoted_tokens.iter().copied().max(),\n                    )\n                {\n                    poa_metrics().epoch_max_drift.set(\n                        i64::try_from(max_tok.saturating_sub(min_tok))\n                            .unwrap_or(i64::MAX),\n                    );\n                }\n                if let Some(max_token) = promoted_tokens.into_iter().max() {\n                    let mut current_epoch_token = self\n                        .current_epoch_token\n                        .lock()\n                        .map_err(|e| anyhow!(\"epoch token lock poisoned: {}\", e))?;\n                    *current_epoch_token = Some(max_token);\n                    poa_metrics()\n                        .leader_epoch\n                        .set(i64::try_from(max_token).unwrap_or(i64::MAX));\n                }\n                poa_metrics().promotion_success_total.inc();\n                poa_metrics()\n                    .promotion_duration_s\n                    .observe(promotion_start.elapsed().as_secs_f64());\n                return Ok(true);\n            }\n            self.release_lease_on_all_nodes().await;\n            let is_last_attempt = attempt_index.saturating_add(1) == self.max_attempts;\n            if !is_last_attempt {\n                self.delay_next_retry().await;\n            }\n        }\n        tracing::warn!(","sourceCodeStart":495,"sourceCodeEnd":531,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/service/adapters/consensus_module/poa.rs#L495-L531","documentation":"In acquire_lease_if_free (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:513), after a successful quorum promotion the adapter locks the shared std Mutex<Option<u64>> current_epoch_token to record the highest promoted epoch. lock() returned a PoisonError: another thread panicked while holding this mutex earlier, so epoch bookkeeping can no longer proceed in this process.","triggerScenarios":"A panic in any code path holding current_epoch_token (promotion success, release, or epoch adoption in has_lease_owner_quorum); the next successful promotion then hits the poisoned mutex and surfaces this error.","commonSituations":"An earlier, unrelated panic inside the PoA service is the root cause; this error is the follow-on symptom. Typically the node keeps failing to produce blocks until restarted.","solutions":["Restart the fuel-core process — a poisoned std Mutex cannot be unpoisoned in-process","Search logs backwards for the original panic that poisoned the lock and fix or report it","Upgrade fuel-core if that panic is a known, already-fixed bug"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_poisoned_lock(e: &anyhow::Error) -> bool {\n    e.to_string().contains(\"poisoned\")\n}","tryCatchPattern":"match leader_lock.can_produce_block().await {\n    Err(e) if is_poisoned_lock(&e) => {\n        // in-process poison: log, flush, and exit so the supervisor restarts us;\n        // the Redis lease TTL safely hands leadership to another node\n        tracing::error!(\"epoch mutex poisoned; restarting process\");\n        std::process::exit(1);\n    }\n    other => other,\n}","preventionTips":["Run fuel-core under a supervisor with automatic restart (systemd Restart=always, k8s liveness)","Alert on the FIRST panic in the poa service, not on follow-on poisoned-lock errors","Never ignore earlier panics in logs — they are the root cause of this error"],"tags":["fuel-core","poa","redis","mutex","panic","concurrency"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}