FuelLabs/fuel-core · critical · anyhow::Error

cannot access epoch token, poisoned lock

Error message

cannot access epoch token, poisoned lock

What it means

release_if_owner (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:952): when quorum ownership is not held, the adapter clears the shared current_epoch_token (std Mutex<Option<u64>>) so a future leader re-promotes from scratch. lock() returned a PoisonError — a thread panicked earlier while holding this mutex — so the token can be neither read nor written in this process and even the release path fails.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:952

            tracing::debug!(
                quorum = self.quorum,
                redis_nodes = self.redis_nodes.len(),
                current_epoch_token = ?self.current_epoch_token_value(),
                lease_key = %self.lease_key,
                "This authority already holds leader lock quorum"
            );
            return Ok(true);
        }
        self.acquire_lease_if_free().await
    }

    async fn release_if_owner(&self) -> anyhow::Result<()> {
        tracing::debug!("Releasing Redis leader lock");
        if !self.has_lease_owner_quorum().await? {
            let mut current_epoch_token = self
                .current_epoch_token
                .lock()
                .map_err(|_| anyhow!("cannot access epoch token, poisoned lock"))?;
            *current_epoch_token = None;
            return Ok(());
        }

        let releases = futures::future::join_all(
            self.redis_nodes
                .iter()
                .map(|redis_node| self.release_lease_on_node(redis_node)),
        )
        .await;
        let released_count = releases.into_iter().filter(|released| *released).count();
        if self.quorum_reached(released_count) {
            let mut current_epoch_token = self
                .current_epoch_token
                .lock()
                .map_err(|_| anyhow!("cannot access epoch token, poisoned lock"))?;
            *current_epoch_token = None;
            Ok(())

View on GitHub (pinned to b9d4d170da)

Solutions

1. Restart the fuel-core process — poisoned std Mutexes do not recover in-process
2. Find and fix the original panic earlier in the logs; this error is only its aftershock
3. Run under a supervisor with auto-restart; the lease TTL expiry hands leadership to another node safely in the meantime

fn is_poisoned_lock(e: &anyhow::Error) -> bool {
    e.to_string().contains("poisoned")
}

match leader_lock.release_if_owner().await {
    Err(e) if is_poisoned_lock(&e) => {
        // poison is terminal in-process: exit and let the supervisor restart;
        // lease TTL expiry releases leadership safely even if release failed
        tracing::error!("epoch mutex poisoned on release; exiting for restart");
        std::process::exit(1);
    }
    other => other,
}

When it happens

Trigger: Any earlier panic in code holding current_epoch_token (release, acquire, or epoch adoption); every subsequent lock on it fails, including this one during shutdown or leadership loss. The Redis lease itself still expires via TTL, but in-process epoch state is frozen.

Common situations: Seen after an unrelated panic in the PoA service; the node keeps hitting this on every release attempt until restarted; the drop_release_guard may still fire a best-effort release on drop.

Related errors

• epoch token lock poisoned: {}
• The lock is poisoned: {}
• Not implemented yet
• The override height is zero. The override height should be g
• At least one redis url is required for leader lock

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/229307dbb9eeac8d. Report an issue: GitHub.