FuelLabs/fuel-core · critical · anyhow::Error

epoch token lock poisoned: {}

Error message

epoch token lock poisoned: {}

What it means

In acquire_lease_if_free (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:513), after a successful quorum promotion the adapter locks the shared std Mutex<Option<u64>> current_epoch_token to record the highest promoted epoch. lock() returned a PoisonError: another thread panicked while holding this mutex earlier, so epoch bookkeeping can no longer proceed in this process.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:513

            );
            if self.quorum_reached(acquired_count) && validity_millis > 0 {
                // Record epoch drift across quorum nodes
                if promoted_tokens.len() > 1
                    && let (Some(min_tok), Some(max_tok)) = (
                        promoted_tokens.iter().copied().min(),
                        promoted_tokens.iter().copied().max(),
                    )
                {
                    poa_metrics().epoch_max_drift.set(
                        i64::try_from(max_tok.saturating_sub(min_tok))
                            .unwrap_or(i64::MAX),
                    );
                }
                if let Some(max_token) = promoted_tokens.into_iter().max() {
                    let mut current_epoch_token = self
                        .current_epoch_token
                        .lock()
                        .map_err(|e| anyhow!("epoch token lock poisoned: {}", e))?;
                    *current_epoch_token = Some(max_token);
                    poa_metrics()
                        .leader_epoch
                        .set(i64::try_from(max_token).unwrap_or(i64::MAX));
                }
                poa_metrics().promotion_success_total.inc();
                poa_metrics()
                    .promotion_duration_s
                    .observe(promotion_start.elapsed().as_secs_f64());
                return Ok(true);
            }
            self.release_lease_on_all_nodes().await;
            let is_last_attempt = attempt_index.saturating_add(1) == self.max_attempts;
            if !is_last_attempt {
                self.delay_next_retry().await;
            }
        }
        tracing::warn!(

View on GitHub (pinned to b9d4d170da)

Solutions

1. Restart the fuel-core process — a poisoned std Mutex cannot be unpoisoned in-process
2. Search logs backwards for the original panic that poisoned the lock and fix or report it
3. Upgrade fuel-core if that panic is a known, already-fixed bug

fn is_poisoned_lock(e: &anyhow::Error) -> bool {
    e.to_string().contains("poisoned")
}

match leader_lock.can_produce_block().await {
    Err(e) if is_poisoned_lock(&e) => {
        // in-process poison: log, flush, and exit so the supervisor restarts us;
        // the Redis lease TTL safely hands leadership to another node
        tracing::error!("epoch mutex poisoned; restarting process");
        std::process::exit(1);
    }
    other => other,
}

When it happens

Trigger: A panic in any code path holding current_epoch_token (promotion success, release, or epoch adoption in has_lease_owner_quorum); the next successful promotion then hits the poisoned mutex and surfaces this error.

Common situations: An earlier, unrelated panic inside the PoA service is the root cause; this error is the follow-on symptom. Typically the node keeps failing to produce blocks until restarted.

Related errors

• cannot access epoch token, poisoned lock
• The lock is poisoned: {}
• At least one redis url is required for leader lock
• Not implemented yet
• The override height is zero. The override height should be g

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/0058a654a4c19487. Report an issue: GitHub.